Community Action Network

Public Science Collaborative conducted surveys, focus groups, and workshops to evaluate data systems' effectiveness. Six human-centered heuristics were identified, including intuitive data entry, accurate reporting, customizable systems, strong governance and security, seamless integration, and adequate training and support. These principles aim to enhance user experiences, improve data accuracy, and ensure system security for stakeholders such as staff, clients, vendors, and agency leaders.

• Data systems
• Evaluation
• Heuristics
• User experience
• Security

ren_fa Follow

Uploaded on Feb 25, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Community Action Network Evaluation of Five Data Systems By Matthew Voss, Cassandra Dorius, Shawn Dorius, and Sarah Walker Heuristics Used to Assess Five Data Systems The Public Science Collaborative conducted two surveys, six focus groups, and three workshops on behalf of the Community Action Network (CAN) to evaluate their current data systems effectiveness and feature gaps. Across these settings, CAN stakeholders were iteratively asked, "What works well for you in your current data system?" And "How might a future system better meet your needs"? The resulting usability principles, also known as heuristics, were then refined into "apples-to-apples" criteria that became the basis of PSC's data system evaluation. To better meet the needs of CAN's target data users and achieve more effective and impactful results, we recommend future systems address these six human-centered heuristics: 1. Intuitive Data Entry: The data system should have interfaces that are easy to use, reducing the chance of errors and improving the overall user experience. Staff shared that options for duplication and formatting alerts are highly valued, as are client portals to support data collection. Vendors remarked that integrating client lists with vendor software is often complicated by inaccurate data, and accuracy would improve with alert features. Clients noted that less confusing, duplicative, and time-consuming entry portals would improve their experience. Meeting the goals of this usability principle would be especially beneficial for staff, clients, and vendors. 2. Accurate Reporting: Reports generated by the system should be trustworthy and reliable, with access to all relevant fields and data. State leaders observed that federal funds require comprehensive and accurate data collection, and misinformation may threaten future funding. ICAA leaders noted that better access to accurate cross-agency reporting was critical to supporting their statewide efforts. Agency leaders reflected that poor reporting was costing local groups money and human resources already in short supply. Staff members shared concerns that reports generated with identical filters, minutes apart, were different, and more trust was needed in findings produced by the data system. Staff, agency leaders, ICAA leaders, and State leaders will be benefitted by improving report accuracy in the new system. 3. Customizable Systems: The data system should allow for agency-level customization and offer tailored options for specific agencies or groups of agencies, giving decision-makers more control over how their local system is used. Agency leaders confirmed that opportunities to customize directly, without engaging data system owners, were particularly valuable. State leaders shared that system-wide customization based on universal reporting needs was also crucial for a successful system. State and agency leaders are particularly benefited from this supporting this usability principle. 4. Strong Governance and Security: The data system must ensure the security and privacy of client data through measures such as privacy certifications, multi-factor authentication, and role-based access controls. Project and employee management are highly valued for creating secure HR opportunities within the system. There was universal agreement about the need for a strong governance and security system among leaders from ICAA, the State, and local agencies, although all data users will be benefited from a secure system, including vendors, front- line staff, and clients. 5. Seamless Integration: Integration with other systems through APIs or bulk data transfer can improve accuracy and reduce the chance of double entry. Integration was highly valued by agency leaders, State program leaders, and ICAA leaders, who seek to reduce data silos, improve access to data insights that drive change, and provide a statewide perspective across multiple agencies and services. 6. Adequate Training and Support: All data system users should have access to role-specific training materials, on- call support, and a formal training process to ensure everyone can effectively use the system. When data system users were asked what type of training they would like to receive, agency leaders told us that training should be asynchronous, 'canned,' and easy to understand to meet the needs of a high-turnover workforce. Front-line staff noted that they would like on-call support to help them address pressing issues promptly. Recommendations for videos, webinars and training manuals were suggested by ICAA leadership, reflecting the need for iterative, formal training that ICAA can help distribute. Staff and leadership from agencies and ICAA would benefit most from implementation of this principle. Evaluation of Five Data Systems 1

2. Community Action Network Evaluation of Five Data Systems Missing Features or Gaps of Current System The CAN data system users we spoke with were concerned about how missing features and gaps in their system prevent them from achieving their desired outcomes or meeting their specific needs. For example, front-line staff reflected that a lack of usable system features made it difficult for them to effectively process new client applications or generate accurate reports. They also noted the critical need for data systems to flag errors and be able to check for duplicate entries within and across systems. Vendors shared the same need for more accurate data, more systematic data fields, better duplicate checking, and more regular updates to resolve errors. Agency leaders described how missing analytic and reporting tools, including client list reports and visual analytics, kept them from identifying critical insights to drive change, and a new system must provide options to customize data collection and reporting to meet local needs. State and ICAA leaders noted that because the current data systems lack integration with other systems, there were more data silos and fragmented views of the information. Further, users described a strong need for improving the client portal for all programs, updating the WAMS data collection tools to improve system reliability, and greater flexibility of reporting across all system fields. Filling these gaps will improve the overall functionality and effectiveness of the CAN data system. By addressing these issues, stakeholders can better utilize the system to achieve their goals, increase efficiency, and gain a more complete view of the information. Additionally, filling these gaps can help prevent users from relying on manual, time-consuming processes to achieve their desired outcomes. Effort/Impact Assessment of Adding Missing Features An Effort/Impact Assessment is a systematic process used to evaluate the potential benefits and costs of a proposed action, project, or investment. It involves analyzing the amount of resources (such as time, money, and person hours) required to implement the project and comparing them to the expected outcomes. This type of assessment is often used in strategic decision making because it provides a clear and objective way to evaluate the potential benefits and drawbacks of a proposed action. By considering both the effort required and the expected impact, Iowa s CAN can determine whether the project is worth pursuing and allocate resources accordingly. This approach helps organizations make informed decisions, prioritize their initiatives, and ultimately achieve their goals more effectively. Below is an Effort/Impact Assessment for the features identified by CAN data system users as important to gain in the new data system. Impact System Integration Reporting accuracy Client portal for all programs Duplicate checking within a system Duplicate checking across systems WAMS system reliability Effort Dashboard and visual analytic tools Reporting for all system fields Evaluation of Five Data Systems 2

3. Community Action Network Evaluation of Five Data Systems Note on Operational Data Stores (ODS): ODSes are designed to integrated data from multiple sources for basic data processing activities such as agency reporting with real-time analysis. PSC was not asked to evaluate an ODS option, but the question of their usefulness emerged in numerous conversations with CAN stakeholders. If pursued, pricing would be determined by RFP to vendors who create ODS systems for government agencies. For example, I-DHHS is currently under contract to build an ODS for early childcare for an estimated cost of 3.5 million. This cost does not include the underlying data systems that feed the ODS. CAP60 Capabilities and Value Description: CAP60 is a well-rounded data system with an average rating of Excellent on the six usability criteria. In particular, CAP60 excels in data entry and customization, with a client portal, an intuitive interface, and extensive options for the customization of programs and client intake. The system also has significant integration with other systems, including ChildPlus, and has set up API connections with other systems where possible or bulk import and export processes for outside systems that do not have APIs. CAP60 provides unlimited training as part of its offerings, including on-call support staff, video tutorials, and one-on-one or group training. The client intake has been translated into over 100 languages and can be flexible to local language needs. Reporting is generally intuitive but flexible, with many canned reports and the ability to create custom reports. CAP60 is used statewide in three states and by many individual agencies in other states. Based on ease of use, CAP60 was our reviewer s top choice for the CAN data pilot. Strengths: This was the best all-around system evaluated, with Excellent ratings in the areas that mattered most to the people we interviewed, including the only system to receive excellent ratings for data entry (straightforward with duplicate and address checking built in), system customization (agency can add fields and questions on their own for no extra cost, and training is provided to learn how to customize the system), system integration (API allows connection with other systems and has established connections with critical partners), and training and support (free, unlimited, and both online and on-call). Among features noted as good : Reporting is simple and accurate, and covers all possible data fields stored in the system. Governance resources include security protocols, encryptions, and role-based access, as well as HR and volunteer management modules available for purchase. Gaps/missing features: Client eligibility is not located on the main household landing page, but is found on a separate tab, which may be inconvenient for users. Custom reports cannot be explicitly saved, although users can re-run prior reports on new data. Cost: Estimated $5,500 - $6,500 annually for an average agency (7,500 households). eLogic Genesis Capabilities and Value Description: eLogic Genesis is a well-rounded data system with an average rating between Good and Excellent on the six usability criteria. Our reviewer found navigating the eLogic Genesis software to be less intuitive than CAP60 because of the wide variety of client assessment options available. On the plus side, these assessments and client intake are customizable and use a client portal. eLogic Genesis is built around the ROMA standards and is especially well-suited for CSBG reporting. Beyond canned reporting options and custom reports based on filters, the system also has Google Looker dashboard options that agencies can purchase at varying cost levels. eLogic Genesis integrates well with other systems, including ChildPlus and WxPro, and will set up scripts or import/export processes for working with outside systems that do not have APIs. eLogic Genesis has limited Weatherization support and recommends WxPro for full Weatherization programs. The client intake has been translated into over 80 languages and can be flexible to local language needs. The base system setup includes creating user guides, and the regular system updates include free training for system users. eLogic Genesis is used statewide in several states and by individual agencies in other states, including one agency in Iowa. Strengths: This system receives Excellent ratings for reporting and integration as the interface is simple to use, all fields are reportable, and an API allows for connections to other systems. Add-on modules allow for data visualization and SQL querying. eLogic is the only HIPPA certified system, with state of the art security controls. They are also the only system we know of to offer data governance training. Gaps/missing features: Limited Weatherization support and reduced LIHEAP functionality compared to other systems. Interface is not intuitive with menus that change based on current screen. Customization and training are fee-for-service and will likely increase overall cost, and costs may fluctuate from year to year based on need for customization and support. Cost: Pricing laid out here and significantly depends on add-ons. $675 a year for base license (CAMP recommended to license at state level for 50+ user discount and set up nesting networks for agencies). Given the apples-to-apples options we estimated, our best guess of an annual agency cost (7,500 households) is $4,500. However, there is a large one-time data migration estimated cost of between $3,000-$5,000 for small agency migrations, $5,000-$10,000 for medium sized agencies, and $50,000-$100,000 for the largest agencies with the most complicated migrations. Evaluation of Five Data Systems 3

4. Community Action Network Evaluation of Five Data Systems EmpowOR Capabilities and Value Description: EmpowOR is a solid data system that performs well, but not excellent, on nearly every criterion, with an average rating of Good across the six usability features. EmpowOR s best capabilities are in the area of system integration, where bulk import and export of client lists allows for some amount of system integration, and the Community Action Data Store allows for a more advanced approach to integration. EmpowOR also has useful client intake processes, with forms that can be customized by agencies and the software provider to be able to work with different programs. Reporting generally appears to be easy to use while maintaining flexibility through creation of customized reports. There is a large amount of onboarding training and training video access for system users. A Weatherization module is in development, but not currently included, and EmpowOR is currently undergoing a SOC 2 Audit and a security compliance process in Michigan. The system provider seems to be especially willing to work with agencies for specific state and program needs. Strengths: EmpowOR received one Excellent rating- for system integration (described above). Reporting is solid, with easy and flexible intake options and duplicate checks for data entry. Reporting features include numerous pre-generated reports as well as options to create custom reports. This system is HUD and DHHS security approved, and training webinars, videos, and supplemental materials are available for use by members. Gaps/missing features: No current Weatherization support and no API in base system. Unlike the CAP60 and eLogic systems we reviewed with dozens of language translations available for intake forms, EmpowOR only provides intake questions in English and Spanish. Cost: Provider declined to give an estimated cost, saying that it depends largely on number of users, add-ons, and contract structure. NIFCAP Capabilities and Value Description: NIFCAP is a capable system with an average rating of Fair across the six usability criteria. CAN staff generally found data entry in the system to be easy as one of the system s better features, but the PSC reviewer considered the icon-based system with pop-up windows to be unintuitive and thought that it would have a significant learning curve for new employees. The best features of the system are training and system customization through direct access to the software creator and designer and that the system was created with Iowa community action in mind. However, the age of the system and one-off customized features for specific agencies have resulted in features and reports that have been deprecated but remain on the system, or features that are designed for use by a specific agency but confuse other agencies users. NIFCAP is currently undergoing a significant system redesign, which may change the system s usability. Strengths: Direct access to the software creator who provided helpful and quick responses to issues was consistently noted as the most positive aspect of data system usage. Additionally, NIFCAP boasts a large number of canned reports. Gaps/missing features: No Weatherization support, incomplete report customization (not all fields available and no bulk export), no API or easy system integration, unused and deprecated reports still in system. Variations in data accuracy noted by users. Cost: Provider declined to give an estimated cost, instead deferring to current ICAA contract. T.H.O. Capabilities and Value Description: T.H.O. was the lowest rated system overall, with an average usability rating between Poor and Fair. Data entry and T.H.O. s user interface were positively rated features among CAN staff who use T.H.O.. However, the system s limitations on bulk import and lack of agency customization on client intake limit the Data Entry rating to Fair . The reporting system has significant user flexibility, but is complicated to use and a bug in the system contributes to inaccurate reports, even occasionally when running the same report with no data updates. The nature of the system also makes it difficult for agencies to customize for their own programs, as large system changes require approval from community action clients across the country, and small changes lead to unused fields for other agencies. Strengths: Data entry is generally simple, with auto-population available from LIHEAP and CSBG online forms. Extensive role-based access supports security. Strong integration with Orion fiscal software. Gaps/missing features: Significantly, T.H.O. had the only critical failure noted in our five-system evaluation. Stakeholders repeatedly described how the system would provide inaccurate and inconsistent reports due to a bug in the system. This coding failure was confirmed in our interviews with the data system provider, and by all accounts would continue unabated as long as the system remains in use. Additional gaps include no Weatherization support, no API or easy system integration (other than Orion), limited technical support, complexity in running reports, and security concerns expressed by CAN staff. Cost: About $6,000 annually, per agency, in current Iowa contract. Evaluation of Five Data Systems 4

5. Community Action Network Evaluation of Five Data Systems Summary Apples-to-Apples Matrix The matrix below summarizes the results of the detailed evaluation of each of the five programs across the six usability features. Ratings are based on a four point scale, with four stars indicating a rating of Excellent , three stars indicating a rating of Good , two stars indicating a rating of Fair , and one star indicating a rating of Poor. A more detailed accounting of the features that led to the rating are provided on the next page. T.H.O. NIFCAP EmpowOR CAP60 eLogic Genesis Data Entry Reporting System Customization Governance and Security System Integration Training and Technical Support Evaluation of Five Data Systems 5

6. Detailed Apples-to-Apples Matrix T.H.O. NIFCAP EmpowOR CAP60 eLogic Genesis Fair Fair Good Excellent Data entry is straightforward, but most comes through client kiosk. Duplicate and address checking across multiple fields. Bulk service assignment. Program eligibility is calculated, but not on main page. Agency has customization, except for federally-required fields. Good Data Entry Data entry is generally simple, with clear household and individual screens. Some automatic duplicate checking and address checking. Auto-population for LIHEAP and CSBG from online forms (CSBG not currently used by Iowa agencies). Icon-based system is less intuitive, and data entry windows that pop up on top of each other are confusing. However, entry is easy within those windows. Some duplicate checking and other input regulation. Data entry is easy, with significant flexibility for client intake, including customization based on program. Duplicate checks on data entry using one or more fields. Agencies have customization ability and provider works with agencies to further customize intake. Agencies have substantial flexibility to create own forms and assessments that clients see in client portal. System works best with CSBG; there is less LIHEAP-specific functionality. Bulk service assignment. Can apply tags to client notes. System interface looks modern but may be unintuitive for some users, including multiple menus that change based on screen or access. Excellent Normal reporting within system. Relatively simple, and all fields are reportable. System is optimized for CSBG reporting. Additional Google Looker Dashboard modules are available for data visualization and SQL querying. Intuitive interface Duplication and formatting alerts Client portal and intake Poor Fair Good Good Reporting Accuracy Simplicity Flexibility Agencies can create reports based on any fields for which data is collected. Accuracy and consistency are issues due to both the complicated nature of running reports and a bug in the system. T.H.O. has been unable to find said bug. A fair number of canned reports and customized reports created for agencies. But agencies cannot access all fields to create own reports. Variations in accuracy and confidence in data depending on data type. Some reports are deprecated or non- functional, but still listed. Fair Less in the way of agency customization, but easy to work with company to get customized reports or features. Reporting appears to be simple, with access to many canned reports and ability to build out advanced/custom reports. Reporting is simple and has access to all fields. Filters are intuitive and usable, with additional canned reports. Custom reports cannot be explicitly saved, but users can re-run previous custom reports on current data. Poor Good Excellent Good System Some ability to customize data entry by removing fields. System is the same for agencies across states, so T.H.O. is hesitant to add features for a specific agency or state unless it s very small. Larger features have to be approved by group of CAA clients across country. Fair Extensive role-based access. Lack of certain security standards prevent system from being used for certain programs. Some project management ability with tracking staff and finances. Provider works with agencies on client intake and work template customization. Agencies also have own customization options in these areas, including staff access and document management. Agency can add fields and questions on own. Most agency-level customization done at no extra cost. Agencies create own program workflows. Works with agencies in onboarding to make sure all programs are covered. Agencies can create their own forms. Costs for new feature development outlined on website. Customization Agency customization Provider customization Fair Good Good Excellent Governance and Security Privacy and security certifications Role-based access Multi-factor authentication Project and No singular certification, but security model contains aspects of NIST SP 800-53, HITRUST CSF, COBIT 5, SOC 2, ISO 27001, PCI DSS, CSA STAR CCM. This sort of structure may be an issue for security compliance in certain programs. Approved HUD and Head Start system. Passes DHHS security standard. System is currently undergoing SOC 2 audit and Michigan security compliance process. Data is encrypted at upload, download, and at rest. SOC reporting with third party security audit. Encryption at all data locations. Role-based access at individual level or by group/role. Human resources module and volunteer management. System built around HIPAA compliance. SOC 2 certification. System uses nested networks to limit access, with additional role- based access within network. System employee and client notifications and can track serve funds. CAMP offers data governance training and consultation. Excellent API that allows connection with other systems. Already has connections with ChildPlus and WxPro. Can set up scripts and processes for bulk manual export and import as necessary for outside systems without API. employee management Fair Poor Excellent Excellent System Integration Deduplication API access Bulk data import/export Fiscal integration Minimal system integration. Strong integration with Orion fiscal software but no other API. Bulk export is possible with extensive CSV reporting. Minimal system integration. At least one agency uses a customized built-in fiscal export to Orion, but no API. Given limits on customizable reporting, there is little ability to mass export. Can bulk import and export client lists. Can serve as single entry point given customization of intake. Community Action Data Store serves as option for API integration with outside systems. API allows connection with other systems. Has established connections with ChildPlus, SSA, and HMIS. Can set up bulk manual export and import processes for outside systems without API. Has worked with some agencies to write checks directly from system, otherwise can integrate with fiscal APIs. Excellent Unlimited training/support, including one-on-one or group training. Video tutorials are available, and there are dedicated on-call support staff. Poor Good Good Good Training and Support Formal training Does formal training with individual agencies and states as needed or requested. Has tried webinars in past. Does not see the point of a manual given the constantly changing nature of the system. Direct access to software creator, with quick response for issues. Yearly on-site agency trainings and yearly train the trainer trainings. Includes webinar training for different levels of users end users, managers, and administrators. User videos and documentation for all levels. User guide development is part of base setup. Free training webinars when software updates. Otherwise, training (webinar or in-person) has costs per hour. Beyond software training, company provides other data training and consultation as well. System set up around CSBG with ROMA standards, has LIHEAP functionality, some Weatherization, but CAMP recommends WxPro if Weatherization support is needed. Business portal. Client intake 80+ language translations. Not a modularized system. process Training manual Training videos Webinars On-call support Client kiosk in development. Not a modularized system. Near rewrite of NIFCAP software will replace current system in Spring 2022. Weatherization module in development. Some agencies currently use for pieces of Weatherization. Client intake Spanish translation, others being developed. Client intake 100+ language translations. No individual user licensing (unlimited system users). Miscellany 6 Evaluation of Five Data Systems