August 16, 2012

Recreating the Wheel

Posted in IT tagged , , , , , at 1:22 PM by Solutions2Projects, LLC

I love documentation and I won’t apologize for it.  Documentation saves time and leaves space to think through the tough issues. 

In IT, documentation is often a four letter word because documentation  isn’t as much fun as playing with technology.  I understand this. 

But, how often have you had to rethink a process or fumble through something that you haven’t done in a while?  How much time was spent on this that could have been reduced if there had been a specification document or work instruction?  What about when key personnel leave and they take their knowledge with them?

Recently I worked with a client on a system that I hadn’t touched in over two years.  Fortunately, I had documented the process and had a work instruction for it and was able to quickly answer the question and address the issue.  If I hadn’t, we would have had to contact support, wait on hold to get through two to three levels of support to get to the answer.  This was assuming it was during the right support hours. 

Another client is struggling to recreate processes performed by technical personnel who have left the company. And, because it was a small IT department, this person was solely responsible for performing some technical tasks and didn’t write anything down.  We are now in the process of starting from scratch to figure out how to get the work done with, of course, a very short timeline. 

The extra time it takes during an initial project implementation to document the system, configuration, and work instructions is less expensive and less time consuming in the long run and not nearly as painful as one might expect.  The documentation doesn’t  have to be formal or complicated; it just needs to impart the necessary information so that others do not have to recreate the wheel.

Advertisements

March 27, 2012

Know Your IT Systems Vendors

Posted in Cloud, Computer Validation, IT, Vendor Audits tagged , , , , , , , , , , , at 9:38 AM by Solutions2Projects, LLC

IT systems and infrastructure are critical to any organization.  This is especially true for life sciences companies selecting and implementing IT systems critical to the business functions supporting compliance functions. Regulatory bodies expect life sciences companies to demonstrate control over these elements regardless of whether they are the ones developing or maintaining the IT systems (infrastructure, software, etc.).

Companies cannot simply toss the responsibility over the fence to the vendors. Life sciences companies are still responsible for the integrity of the data and control over the systems.  They may delegate but only after verifying the vendor can meet the compliance and control requirements. 

This is where vendor audits come in to play. 

Vendor audits for software are not new.  Over the past decade I’ve seen the importance of vendor audits for software wax and wane and wax again.  In light of the increase in cloud and hosted solutions chosen by companies to decrease overall spend, the need for vendor audits is critical.

And, as biotechs become more virtual and more services are outsourced (CRO, CMO, data management, complaint handling, etc.), it is imperative that companies verify their vendors meet compliance requirements as well as their own procedural and process requirements.   The vendor’s IT systems and controls must meet the requirements as if they were hosted by your own company.  Not all vendors perceive the need to meet compliance requirements at the same level and you need to know before you enter any agreements.  Once you’ve signed the contracts, you’ve lost your leverage for process improvements and controls. 

Why conduct the audits? 

  • Gain high level of confidence that the computerized system will meet technical, commercial and regulatory requirements (GAMP 5)
  • Confirm the supplier builds quality and integrity into the software product during development
  • Leverage knowledge, experience and documentation of supplier (GAMP 5) to potentially reduce validation effort
  • Confirm processes and controls when  outsourcing IT / software functions (SaaS, PaaS, IaaS, hosted solutions, co-locations)

When should audits be performed?

  • For high risk systems / outsourced services
  • Before any contracts are signed!
  • Scheduled follow up audits based on
    • Audit results
    • External audit program
    • Risk assessment
    • Significant vendor business changes
    • When there are issues with the vendor

How are audits performed?

  • Similar to other vendor audits for CMOs or other critical suppliers
  • Plan for the audit and communicate expectations to the vendor
  • Conduct the on-site audit (for IT systems, Quality and IT representatives should participate)
  • Summarize findings with the vendor at the end of the audit
  • Document findings in an audit report and provide to the vendor for a response
  • Follow up on observations and document

The financial cost of, and risk associated with, software solutions has increased exponentially which means that it is imperative for organizations to understand what they are getting into before they sign on the dotted line.  The cost of a software or IT system blunder can be expensive in terms of resources, time and can make or break a life sciences company.  If you cannot demonstrate control, and therefore the integrity of your data, for systems supporting drug product administered to patients, a regulatory body may not grant approval for your product or could shut down manufacturing operations.  Your company owns the data and the responsibility even if it service is outsourced. 

Knowing your IT vendors gives you the knowledge to reduce the risks associated with the IT solutions in your life sciences company.  Without this knowledge, you are powerless to defend your risk assessment and risk mitigation strategy to regulatory agencies.

February 2, 2012

Nail Down the Details Before Signing Any Agreements

Posted in IT, Project Management tagged , , , , , , at 10:47 AM by Solutions2Projects, LLC

Negotiating contracts with software vendors is always a challenge.  The vendors won’t commit to resources until contracts are signed but without knowing the resources and their qualifications and availability, how do you know you will get what you want?

Vendors also want to defer detailed planning until after the agreements are signed.  Once again this is at odds with best practices from a planning perspective.  During the planning, a lot of details come to light that affect the final buying decision and overall timeline and budget.  Signing before nailing down these details generally ends up being very costly from an expectation and overall resource perspective. 

 A number of years ago I was working for a biotech company and we were selecting an ERP system.  A big name player was interested in breaking into the small-midsized life sciences market and saw our company as an opportunity to make this happen. Our budget wasn’t in line with their typical installs and they were touting a turnkey solution for life sciences.  I was skeptical.  They claimed the implementation and validation could be performed within our budget.  After detailed planning and working through the resource assumptions, it became very clear that they were shifting the responsibility for significant documentation tasks to our team (which was limited) which resulted in the reduced consulting fees.  Once we shifted the responsibility for those tasks back to their consulting team (to meet our timeline), the cost went through the roof and was no longer feasible.  This was not a surprise to us.  What was surprising was that they thought they would slip it by us. 

 In other cases, I’ve been brought into manage projects after agreements have been signed and quickly realize that my client made a series of assumptions (not documented) that were not aligned with what the vendor planned to deliver.  This tends to put me in the awkward and uncomfortable position of renegotiating the contract without much leverage as my client has already committed.  It often requires significant diplomacy on my part in that I have to demonstrate what was overlooked by them during the selection process.  Since they tend to have to defend their decisions and actions internally (politics!) and the scope or budget or timeline changes once detailed planning is performed, it can get pretty hairy. 

 If you are able to work through the details before signing the agreement, a lot of this messiness can be avoided or minimized.  I recommend performing the detailed planning prior to signing contracts, working with the vendor to create detailed statements of work to minimize surprises, and generating a project charter capturing the project elements.  The project charter is an internal document but I generally have the vendor consultants read and sign as well in order to hold them accountable as members of the project team. 

 And one of thing greatest benefits of going through this process is that you can learn very quickly how committed the software vendor is to the success of your project and to your company.  If a vendor is completely resistant to investing the time to generate a detailed statement of work based on detailed planning, this is a major red flag.  I would seriously reconsider doing business with that particular vendor.  Detailed planning sets everyone up for success.

January 31, 2012

IT Deviations: Tool for Improving Overall Performance

Posted in IT tagged , , , , , at 7:36 AM by Solutions2Projects, LLC

I am a firm believer in documenting deviations from performance expectations for IT systems.  This includes functional and procedural errors, both system and human.  Documenting deviations provides data for better management and control over IT systems.   The process for capturing and managing IT deviations should be simple to be most effective as we know most IT folks do not like cumbersome documentation tasks especially if they think they will be penalized as part of the process (which should not be the case, ever!). 

 A basic deviation has the following elements:

  • Deviation description
  • Date(s) of occurrence
  • Categorization for tracking purposes (planned, unplanned)
  • How it was discovered and by whom
  • Investigation into source of deviation
  • Recommended resolution or actions to be taken
  • Approvals of investigation and recommended resolution
  • Documented action taken to resolve issue
  • Approval of action taken

 Deviation examples include:

  • Backup failures
  • Failure to follow documented procedures
  • Unexpected system behaviors
  • Unplanned outages

By tracking and trending the deviations, you can see if there are recurring issues and answer the following questions to improve data integrity and reliability. 

  • Is additional training required?
  • Should work instructions be developed or be more detailed? 
  • Is there a technical issue that needs to be addressed due to a repeated failure? 
  • Is there a configuration issue that needs to be addressed?
  • Should the issues be addressed with a vendor? 
  • Does the system need to be replaced?

 IT personnel and business system owners gain visibility into issues and can track patterns if deviations are documented.  It is too easy to forget when something happened (let alone if it happened at all) and how it was resolved.  We have too much to keep track of these days.  Deviations are often seen as something that is bad rather than as a useful tool to gather information to improve overall performance.  For those of us in IT that like data, deviations should be seen as an effective tool to gather data to improve overall system performance.

January 30, 2012

IT Policies to Establish Control

Posted in IT tagged , , , , , at 7:33 AM by Solutions2Projects, LLC

When we at Solutions2Projects are brought into a biotech or medical device company to work for the first time, it is usually to provide selection or implementation services for the first big IT project.  The companies are usually positioning themselves for an IPO and need greater control over their financial transactions or are preparing for commercialization and therefore need a system for inventory management or manufacturing.  In both cases, the companies need policies and procedures for managing these systems once in place. 

 Over the years we have identified the following as the minimum policies that need to be in place once the system is released into production.  If the system is a GxP system and is being validated, the policies are verified as part of the validation process. 

  • Change control / change management
  • Operation and maintenance
  • Security
  • Passwords
  • Backup and restoration
  • Deviation management

 Having these policies in place (and adhering to them) provides IT personnel, affected system users, and business owners with a foundation for control.  For GxP systems, FDA has been known to hand out 483’s for GxP systems that were not controlled from inception.  Think about it.  If you are basing a submission on data from a system that was not under control from inception, how can you demonstrate data integrity and data reliability?  Getting the systems under control from the beginning reduces the system and data integrity risks. 

 Of course, the policies need to be followed.  There’s no point in creating policies and ignoring them.  IT needs to embrace these policies to be successful.  Getting IT involved in the generation and release of these policies increases the likelihood of adoption and success. And the policies need to be in a centralized location for all affected personnel to access (not just IT).  This is generally handled by Quality Assurance (QA) as they are used to document management.  If a training program is in place for GxP documents, these should be incorporated into the training program as well.  

 For each system, there will be procedures to define how the policy requirements are met.  A few examples include

  • the periodic review of users and access
  • instructions for performing test restorations and verifications
  • periodic maintenance activities for the system (technical, application)
  • periodic review of failed logins

 Each system will require different instructions for accessing the data or performing the tasks.  The procedures or work instructions allow the IT personnel to focus on the data or value-added activity and not struggle with how to get the data or perform the task.  This is especially true for activities that are performed infrequently, like annual restoration tests. 

 The policies and procedures can be expanded on over time.  Initially, enough details need to be included to meet compliance requirements and provide affected personnel with guidance to achieve the desired level of control. 

 I understand that ‘control’ is a four letter word to a lot of IT folks.  But, control reduces risk and increases data integrity and reliability, which is what IT is tasked with ensuring.  Compliance (SOX, GxP) mandates establishing control over these systems, and policies and procedures are one of the tools companies can use to satisfy the requirement.