Requests for bulk data
From district court case information systems
March 12, 2003
There is an increasing number of requests from credit reporting services, abstractors, and other such entities for a copy of all civil judgments, criminal convictions, all evictions etc.
In the past it has been our practice to fulfill these requests via a contractor. We obtained the specific request details from the requesting entity and then had the contractor prepare a cost estimate for fulfilling the request according to those details. The estimate was returned to the requesting entity and if they formally agreed to pay the estimated amount directly to the contractor, the contractor would fulfill the request. The average estimated cost to fulfill each “custom request” was approximately $1200.
This method provided the requestor with the desired information while not consuming development staff-time to fulfill the requests.
Prior to 2002, there had been very few requests and even fewer actually fulfilled.
In 2002 there was a significant increase in the number of requests received and in the willingness to spend $1000 or $1500 to obtain the data. Furthermore, all recent requests contained an additional request to receive “regular” updates. This is something we had never done in the past.
To accommodate the increasing number of requests, I propose “selling” a subscription to a web site. Located at that web site would be files available for download for the criminal conviction database and the civil judgment database. The subscriptions would be sold on an annual basis at a cost of $2400 per year. The files on the site would be updated monthly.
Much of the technical infrastructure to complete this concept is currently in place via the data warehouse.
In arriving at the above mentioned proposal, consideration was given to several alternative methods, including:
1) Providing, at a cost, a web site much like the warehouse where companies could search for persons of interest. However, because these companies do many searches in many states, they would like the data to reside “in their system”, as part of an aggregated database. Because of the desire to include the data within their database, I do not believe the response to this method would be worthwhile. However, if this method were to be pursued, the technical infrastructure that would be necessary is largely in place via the data warehouse.
2) Providing the entire database at a cost, then providing ‘regular’ updates to the data. While this would minimize the amount of data transferred between the courts and the requesting entity by only sending “changed” or “added” records, it has other drawbacks. One of the largest drawbacks is the deletion of records. If a record is deleted from the system, how do we notify the requesting entity it was deleted? Rather, by providing the entire database each month, the requesting entity will need to purge all ND data from their system, then re-add the current data. This is a process many are they are familiar with and willing to do.
3) Do nothing with the requests. While this is the easiest, it will be increasingly difficult to avoid the requests. However it is certainly a valid option.
I have contacted others across the state and country and asked what their practices are. Following is a brief listing of some of the responses. As the responses indicate, the only consistent themes are that everyone is seeing an increase and everyone is struggling with how to handle these requests.
Does the North Dakota Department of Transportation “Sell” data:
Yes. If the requestor meets the Driver Privacy Protection requirements as found in NDCC 39-33. At this time the cost is $9 per 1000 names.
Any other type of request is sold at the going rate of $3 per record.
Does the North Dakota Secretary of State’s Office sell records or access to records?
Yes, for the most part, all of the data collected in our office is available and we do sell it. CT, Lexus Nexus, financial institutions, buy and access our data. Other state agencies also access are data and many tie right into our AS400. For example, Tax, PSC, Workers Comp, etc.
We currently provide most information to requestors without giving personal identifiers (SSN, DOB, Licenses, etc.) We do bulk downloads of civil and landlord/tenant data.
Our fees are based on time and materials only. We charge $14 per 1/2 hour staff time and $25 per 1/2 hour processing time.
Current requestors pay around $50 a month for weekly or twice weekly downloads
Utah provides search capabilities for a fee to companies, newspapers, credit bureaus, etc. and free to state agencies, via an online case query system. Additionally, a database can be provided but certain individual identifiers will be eliminated. However, over the last 2 to 3 years I have not received a request for the data base.
Our online query system "XCHANGE" user requests continue to grow. Currently, we have appropriately 600 paying customers, 180 non-paying customers, and we average about 14 new customers per month. The XCHANGE system provides them a single case report. For the paying customer it cost the following:
$25 for the setup fee,
$30 a month providing a 120 minutes access
.10 a minute for over 120 minutes
$25 for reactivation.
Additionally, images of our judgments are posted and can be downloaded by our customers.
Utah Courts are guided by Utah Court Rules - Rule 4-202.12 Access to electronic data elements.
The rule intent:
to provide the extent of access to data elements maintained in a computer data base.
to protect the right of access by the public to information regarding the conduct of court business.
to protect privacy interests from intrusion made possible by the increased accessibility of information recorded, stored, and transmitted in an electronic medium.
to protect the independence of the judicial decision making process from undue influence due to the release of court data.
Utah Administrative Office of the Courts has a Information Services department which provides information internally to the courts (data warehousing, and court matrix) and manages the XCHANGE system and customers.
North Carolina Courts
In North Carolina, we currently charge to provide batch extracts to private entities. The charge includes costs for analyst and programmer time, administrative time, cpu run time and a transfer medium (cd or ftp) charge. Some companies are more than willing to pay the charges, other companies have repeatedly challenged us on our charging method.
All requests are honored based on availability of staff at the time of the request. Some customers are unhappy when we cannot honor a request due to limited staff. They sometimes elevate their requests all the way to the director of the AOC. Staff time is spent handling these situations.
We are not able to provide the 'entire database' for either the criminal or the civil system because the run time to separate public from non-public information would adversely affect production systems. AOC attornies spent a great deal of time defending this position when one customer hired and attorney to argue their right to get a copy of the entire criminal database.
We have concerns about continuing to provide even partial extracts. It has come to our attention that some private companies are accumulating this data to build their own databases and are selling the data. These private databases could contain old or corrupt data.
Our policy for providing batch extracts of criminal and civil data will be reviewed over the upcoming months.
Arizona is also experiencing an increase in requests. We have a committee defining access rules.
We are also investigating the creation of a Data Mart off of our Data Warehouse that we be accessible only to paid subscribers and would allow them to run reports that they write or download data that they select (and program the transfer of). This would get us out of the custom extract business and hopefully encourage everyone to keep the data up-to-date (since they are paying in advance for a one year subscription).
We also get frequent requests for data from both systems and have developed a packet of materials for each that is distributed to the requestor. The packets explain the cost to them and what data will be provided. They are also told that they can purchase updates to the data that are provided monthly. For criminal/motor vehicle data, we provide only disclosable conviction data (no pending cases and no non-conviction data) and we require that they sign-off on a statement about the data they will be receiving.
[In the example provided, the civil request cost was $119 and the criminal/traffic cost was $400]
After 3 years of discussions and rule-making around these issues, the Vermont Supreme Court decided to "just say no".
Our "Rules Governing Dissemination of Electronic Case Records" states in part:
"Because these rules provide public access on a case-by-case basis, the judiciary does not provide electronic case record compilations, either in electronic or printed form, unless a compilation is an electronic case record report made publically accessible by § 5. In enabling public access to electronic case records pursuant to this policy, the court administrator shall ensure that no person may obtain an electronic case record compilation. The court administrator may waive this policy pursuant to a data dissemination contract governed by § 6 of these rules."
§ 5 refers to our case management system reports: if we're using a report ourselves and it does not include restricted information such as juvenile records or personal ID numbers like SSN's, we will give it to a requestor.
§ 6 refers to contracts we may write with "public purpose agencies" (a phrase I believe I stole from the Washington courts electronic records policy). No commercial agents need apply.
We do plan on providing case access on a case by case basis through our data warehouse in late fall.
Rhode Island Courts
The State of Rhode Island’s Unified Court system runs an Oracle based criminal & traffic case management application from SCT (now ACS) We provide a standard set of data files to a requestor in ASCII on CD (Data file formats attached) Our state statutes on access to public information allows us to bill “reasonable costs” for the data. The first company to order the CD was charged $10,000 and gladly sent the check even before we had completed the file. The charge was based on an hourly rate of $75/hr which was the average between a state employee’s hourly rate and the Oracle consultants we are using times the hours to produce. (you can do the math)
For subsequent CD’s we’ve been charging $375 because now it’s just a run job and not a development job. We are considering raising the $375 to $500 to allow for administrative time.
The Rhode Island Civil application system is currently running on a WANG VS (scheduled for replacement this year) We provide the entire file to anyone who asks for it @ $300.
Keep in mind we are now re-thinking this strategy. If the private sector is purchasing this data and then turning around and selling their services, these fees may be cheap, cheap, cheap.
We do not give out bulk data by Supreme Court rule--we do give real-time access on a case by case basis.
The Alaska Court System is a statewide system with 28 court sites around the state. Each court has a UNIX computer system and sends a DAT tape of their systems to Administration at the end of each quarter. We load these files into an onsite computer to do statistics and we do create an index tape for anyone who wants it. The requester has to provide the tape. There is no cost to the requester but we do not provide any other data sources, programming or additional fields of information.
New Mexico Courts
We here in New Mexico we offer a Case look-up application for the public. It is not a batch report it is based on a case number or name. We do not sell data to companies
Puerto Rico Courts
In Puerto Rico our policy is to provide the same access we provide to the average citizen. They can develop any tood thay want (robots, etc.) to automatically search our online service, but thats about it. If every company that needs this info asks for a specific media/data set/output type we would be working for them.
The services we provide are for our citizens, not for companies that will profit from our work.
Larry Webster of SEARCH responded with:
We dealt with this issue in Delaware, as have many other states. We generally denied access to entire databases on privacy grounds. People could have anything they wanted, but not everything. In other words, we would give them individual cases, but not aggregate information. There have been a number of federal and state appellate court decisions in this area, though the law does not seem to be settled yet. States have taken a variety of positions on this and related issues.
We would give our databases to anyone who wanted them, if we stripped personal identifying information, like names, case numbers, etc. This allowed researchers and the media the opportunity to monitor the activities of the judicial branch without causing problems with the privacy rights of individuals.
Danial Hall of the National Center for State Courts
A recent emailing on a nationwide court administrator’s list server had the following comments from Daniel Hall of the National Center for State Courts. The posting was in response to the question:
“We are about to make detailed abstracts of all cases in all trial courts (save one) available to the public on the web. We have had a similar, but limited function in place for a good while. Our current service is free the new one will be fee based, I expect issues on that score.
My inquiry to the rest of you who may have undertaken similar endeavors is:
What other bad stuff might I anticipate?”
Daniel Hall’s response:
I don't know how detailed your "abstracts" of cases are but I have some experience in the area of public access to electronic court records. When I was with the Colorado AOC I led the project that established the policy governing the release of electronic access of court records to the public. There are all sorts of land minds. A brief list:
1. What records are released. Some of this is easy (e.g. victims/witnesses names addresses, social security numbers etc are not released) other fields are difficult or are in grey areas (names of parties of divorce used by realators and debt collectors, some juvenile information, sentencing info, etc.)
2. Who is the custodian of the record. Statutes usually state that the clerk of court is the custodian of the paper record but the electronic record is typcially a state wide issue and so the AOC becomes the custodian of the record. This has some liability issues.
3. Bulk data. If the data is released in bulk than it can be combined with other data bases that vendors have to merge with social security numbers, etc. This can circumvent privacy policies that the AOC has adopted in this context.
4. Quality of data. The release of inaccurate data has liability issues. We had one instance in Colorado where there was a law suit against a Vendor who used bad court disposition data and that resulted in an applicant not being considered for a job. We had a number of complaints lodged with the AOC over this same issue.
3. Financing. Colorado contracted through a bid process with a private vendor who mirrored the data base and provided the approved data elements to the public on there own system, which was fee based. The Judicial Deparmtent was sued (and I was named personally) in federal court by another vendor for uncompetitive practices. I will spare you the gory details but the suit was ultimately dismissed. The point being that this information is very valuable to vendors in a very competitive environment. By the way, this system has generated significant dollars to the vendor which might be a source of revenue to the AOC.
Finally, the Conference of State Court Administrators and the Conference of Chief Justices endorsed two documents.
The first is a lengthy discussion of the process of establishing a policy entitled: Public Access to Court Records: Guidelines for Policy Development by State Courts. This document is attached to the email as ATTACH3a.
The second is a much smaller whitepaper on the subject entitled: White Paper on Court Leadership in Justice Information Sharing. This document is attached to the email as ATTACH3b