Data automation and web scraping
Collecting and preparing data automatically
In many departments the week looks similar: someone opens the same websites or portals, opens PDFs, copies numbers into a spreadsheet and checks whether anything has changed since last time. The information exists – but it lives in different places and in different formats.
Such research and transfer work can be automated in most cases. A program retrieves the sources regularly, reads out the relevant details, checks them and delivers them in structured form – in Excel, in a database or directly in a report.
Web scraping is just one of several techniques. Which one is used depends on the source.
Typical starting points
- Prices, availability or master data of suppliers and competitors are looked up by hand on their websites on a regular basis.
- Figures from published annual financial statements, registers or government portals have to be picked out company by company.
- PDF reports or invoices arrive every week and individual values are transferred into a spreadsheet.
- Data from an API should be fetched regularly and reconciled with internal files.
- Several exports from different systems are merged by copy-and-paste before the actual evaluation can begin.
Who is this for?
Businesses and departments where staff regularly gather information from external or internal sources: purchasing, sales, market monitoring, controlling, consultancies, administration. What matters is not the industry, but that the research is recurring and follows a recognisable pattern.
What can be automated
Websites and portals
Retrieve publicly accessible pages regularly and read out the relevant details – prices, product data, publications or register entries. For portals with a login only where the terms of use allow it.
APIs
If a provider offers an API, that is almost always the better route: more stable, faster and explicitly intended for it. I connect APIs and handle authentication, rate limits and error cases.
PDF documents
Read values from reports, invoices or annual statements automatically – tabular content as well as details in running text. For scanned documents, text recognition is added.
Excel, CSV and text files
Automatically read, standardise and merge regularly arriving files, even if structure and column names vary between deliveries.
Databases and exports
Query data from internal systems and combine it with external sources so evaluations no longer have to be assembled from several files.
Validation and delivery
Every pipeline checks the retrieved data for completeness and plausibility and stores it where it is needed: as an Excel file, in a database or as the basis for reporting.
How I decide which technique fits
The starting question is not whether web scraping is possible, but which source delivers the information most reliably. If there is an API, I use it. If there is a structured export, it is usually better than reading a website. Only if both are missing is the page itself read.
Implementation is usually done in Python, because retrieval, validation, processing and storage can be combined into one maintainable workflow. For your team the technology behind it is secondary: what matters is that the data arrives on time, checked and in a usable format.
When does it make sense?
Makes sense when …
- the same information is gathered regularly from the same sources.
- the sources are technically accessible and their use for this purpose is permitted.
- manual gathering is the bottleneck before the actual evaluation.
- the data is to be processed further in Excel, a database or a report.
Less suitable when …
- the research is a one-off – then manual work is often faster than any automation.
- the source explicitly prohibits such use, or personal data is involved whose processing would not be permitted.
- the source data is so unreliable that every delivery would have to be checked by hand anyway.
How I work
- 01
Clarify sources and needs
Which information is needed, from which sources, how often, and what happens with the data afterwards? I look at the sources and check whether and how they can be retrieved automatically.
- 02
Check feasibility
Before a quote is made, I test the critical sources. That way you know in advance whether the automation holds technically and where limitations exist.
- 03
Build the pipeline
Retrieval, validation, processing and storage are implemented step by step and tested with real data. Every run is logged so deviations become visible.
- 04
Operation and handover
You receive the source code, operating documentation and a walkthrough. On request I set up the scheduled execution and accompany the first runs.
What you get
- An automated data pipeline with logged runs
- Structured output in the required format – Excel, CSV, database or reporting basis
- Plausibility checks that flag missing or suspicious values
- Source code, operating documentation and a walkthrough
Limits
- Not every website can be read automatically. Technical protection, terms of use or legal requirements can stand in the way. I check that in advance and tell you openly.
- Sources change. If a website restructures, the pipeline has to be adapted. I build it so such changes are detected early – they cannot be prevented.
- The quality of the results depends on the quality of the source. Automation can make incomplete or contradictory source data visible, but cannot correct it.
Frequently asked
- What is web scraping?
- Web scraping is the automated extraction of information from websites. A program opens the pages, recognises the relevant details – prices, addresses or table values, say – and saves them in structured form. It replaces manual copying from the browser but changes nothing on the website itself.
- Which data sources can be processed automatically?
- Websites and portals, APIs, PDF documents, Excel and CSV files, databases and structured exports. APIs and exports are the most reliable because they are intended for machine use. Websites are possible but more prone to changes.
- Can data be read from PDFs automatically?
- Yes. With digitally generated PDFs, text and tables can be read directly. With scanned documents, text recognition is applied first and its result has to be checked. How reliably that works depends on the structure of the documents – consistent templates are unproblematic, highly variable documents need additional checks.
- What happens when a website changes?
- Then the pipeline delivers either no data or suspicious data. That is why every run checks the result and reports deviations instead of silently passing on wrong values. Adapting to the changed page is then usually a small change that either I make or your team makes using the documentation.
- Is web scraping always possible?
- No. Some websites prohibit automated reading in their terms of use, others protect themselves technically. For personal data, data protection rules apply in addition. I check this before every project and propose a different route where necessary – often there is an API or an export that is better suited.
- Can the data then be delivered to Excel automatically?
- Yes. The pipeline can generate finished Excel files, fill existing workbooks or provide a data source that Excel retrieves via Power Query. The familiar working environment stays, only the gathering disappears.
Next step
Which information do your staff keep gathering over and over?
Name the sources and what the data is needed for. I check whether and how retrieval can be automated – and tell you if it can't.