Trusted by Market Leaders in Education, Travel, Finance and E-commerce since 2007
We put excellence, value and quality above all - and it shows
A Technology Partnership That Goes Beyond Code
“Arbisoft has been my most trusted technology partner for now over 15 years. Arbisoft has very unique methods of recruiting and training, and the results demonstrate that. They have great teams, great positive attitudes and great communication.”
How Arbisoft helped AEU reduce crawltime from 168 hours to less than 4 hours
Arbisoft helped Advanced Energy United (AEU) reduce its data crawl time from over a week to less than 4 hours, using a custom-designed distributed crawling mechanism and intelligent data extractors capable of semantic analysis. This resulted in a 97.6% decrease in minimum runtime duration and significant cost savings. Arbisoft enabled AEU to provide updated information to their subscribers more quickly and make better data-driven decisions.
IndustryEnergy
HeadquartersBoston
PlatformWeb
“We now have more data than the largest online encyclopedia as a result of leveraging Arbisoft’s expertise. They’re definitely web crawling experts.”Eric FitzVice President, Engineering and Product Development
AEU & Arbisoft
Advanced Energy United or AEU is a group of businesses on a mission to nurture a prosperous economy powered by secure, sustainable, clean and affordable energy. They do this using policy advocacy, analysis and education — and to do that they need access to data. A lot of data. AEU’s product, InsightEngine, collects vast amounts of public data from Public Utility Commission websites for each U.S. state and then provides it to InsightEngine subscribers in an easy-to-understand aggregate they can use to obtain updated information, track long-term trends, and most importantly make better decisions. And this is where Arbisoft comes in.
aee - How we fit together
To be competitive, InsightEngine required daily data crawls of 50 large PUC websites. This entailed massive overhead costs in time, performance, and API service expenses. There had to be a more efficient way of doing this–and they came to Arbisoft to find it.
We implemented a custom-designed distributed crawling mechanism that used automated intelligent data extractors capable of semantic analysis. We optimized their system and added the ability to clean up data in realtime, further cutting down time between data collection and data usability. We also provided a REST API that fed data to their user-facing Ruby on Rails app. The high performance API was initially implemented using Google App Engine and later transitioned to the Django REST Framework app.
