Week 4 is about getting data that does not exist yet in a form anybody can use. That is four different skills wearing one name: telling the shapes of data apart, knowing what mining actually is, choosing how to collect, and writing down what you collected so the next person can trust it.
Six sections. Read the section, then do the drill under it. The drill is the part that counts, and it is scored the first time you clear it.
Nothing cleared yet.
Structured data has fixed fields and one meaning per field. A table, a CSV with a header row, the asset register. You can ask it a question without reading it.
Semi-structured data carries its own keys but not a fixed shape. JSON, XML, a log line that starts with a timestamp and ends with a sentence. A machine can find its way around, but it has to survive the field that is only there sometimes.
Unstructured data is everything a person has to read. The photo of the label, the email, the PDF, the ticket description.
The trap. The label describes the container, not the truth inside it. A serial number typed into a ticket description is perfectly structured information sitting in an unstructured place. Most of the work in week 4 is moving true things out of unstructured places into structured ones, and losing none of them on the way.
Gathering is getting the data. Mining is finding a pattern in data you already hold that nobody put there on purpose. They are different jobs, and the second one only works if the first one was done honestly.
Four families cover almost everything you will meet. Classification predicts a label you already have examples of. Clustering finds groups you did not define. Association finds things that turn up together. Anomaly detection finds the row that is unlike its neighbors.
Two warnings, and the second one is the one that gets people.
A pattern is not a cause. Devices bought in the same month failing in the same month is a pattern; it does not tell you the month caused it.
Mining a small, dirty table mostly finds your own mistakes. If one person filled a field by hand for a year, the strongest pattern in that column is that person, not the fleet. Clean first, mine second, and say which you did.
This week you learn to scrape. That makes scraping the tool you reach for, the way a new spanner makes everything look like a bolt. It is the last route on the list, not the first, and knowing why is worth more than the code.
Read this before you write a selector this week.
Do not point a scraper at a vendor warranty portal with our serial numbers in it. Not Lenovo, not Dell, not any of them. Three people automating a supplier site, from a corporate network, against company assets, is a conversation with that supplier and not a training exercise. For the real assignment, use the portal by hand or ask for the supported export, which is exactly what section three is for.
Practice the technique somewhere published for the purpose instead. toscrape.com is a sandbox that exists to be scraped: a fake bookstore and a quotes site with pagination, JavaScript-rendered pages, a login and a table layout, so you can meet every awkward shape without touching anybody's real service. Checked 29 August 2026.
Before the first request. Read the terms of use. Read robots.txt and understand what it is: a stated preference, not a wall. It cannot stop you, and ignoring it is on the record. Identify your client honestly. Ask whether the data is personal, and whether we are allowed to send our serial numbers to a site we do not control.
While it runs. One request at a time, with a wait between them. Cache every page you fetch, so a re-run of your parser does not re-run your traffic. Stop on the first sign the site is unhappy rather than pushing through it.
Pick your own tool. The plan names Python with BeautifulSoup or Scrapy as an example, and it is a good starting point. Node with cheerio, PowerShell, Power Query, SQL or a notebook are all fine if they get the job done. Everything on this page is true whichever you choose, because none of it is about the language: bound the request, pace it, cache what you fetched, follow the pages, fail loudly on a missing field, and write something structured out.
And if the vendor publishes an API and you use it instead of writing a scraper, you have not dodged the exercise. You have given the better answer, which is what section three is about.
Do this part with your hands, not by reading. The Selector Bench gives you six pages held as text, a box to type a real CSS selector into, and the exact list of what came back. Stage four redesigns the site under you and your working selector stops returning anything, which is the lesson below and the one no tutorial can stage on demand.
Expect the selector to break. Class names are cosmetic and change with a redesign. Anchor on something meaning-bearing where you can, and assert what you expect: if the field you need is missing, the run should fail loudly rather than write an empty string.
Parsed is not true. If the page shows a dash where the warranty date should be and your parser hands back an empty string, that is not "no warranty". It is "unknown", and the difference is the whole assignment. An empty cell that means unknown, sitting next to an empty cell that means none, is how a register quietly starts lying.
The plan asks you to collect a dataset and document the process. The document is not a write-up you do at the end. It is a log you keep while you work, and it is the part that makes your data usable by somebody who was not there.
Seven columns. One row per field you collected.
Column five is the one people fake. Attempted is what you asked for, not what you held. Returned is what came back with an answer. Everything between the two is your miss list, and it belongs in column seven with a reason. A log that reports only the successes reads as full coverage to the person who has to sign the number.
Column six is the one people skip. "Where the portal gave a range I took the later date" is an assumption. Written down it is a method. Left out it is an error somebody finds in month nine.
Collecting is the start. Three more decisions decide whether the data is still worth anything in six months, and the plan names all three.
Warehousing is what we keep and for how long. The live register answers what is true now; it cannot answer what was true in March unless somebody decided, in March, to keep a copy. Retention is a decision with a cost, and forever is not a policy.
Integration is how two systems are joined. A join needs a key with one shape and one owner. The register calls it Asset Tag, a vendor calls it Service Tag, and the mapping between them has to be written down or it gets re-invented differently by the next person.
Synchronization is what happens when they disagree. Every field needs one system that wins, and a rule for the rows that do not match.
We have this problem right now, and it has a number. On 21 August 2026 Jira Assets held 615 laptops and about 34 PCs, and Intune saw about 843 Windows devices. Those are not the same fleet counted twice, they are two systems with different rules about what counts and when a record leaves. Identity completeness sat around 93.5 percent, which is the measure of how far apart they are.
The date is part of the number. A week later, on 28 August, the same export held 627 laptops and 36 PCs. Nothing was wrong with the first count and nothing is wrong with the second. A register is a photograph of a date, and a count quoted without one is a claim nobody can check.
Nobody in the lab called that a synchronization problem until now. It is one, and it is the reconciliation you will meet in week 5.
The first five are the ones printed on your plan. The rest are here because those five leave gaps, and a gap in a reading list is a gap in what you know. Three of the five carry video, so you can watch rather than read if that suits you better.
Every link was opened and checked on 29 August 2026. Each card says what the resource is good for and, more usefully, what it does not cover.
The full course, taught by Joe Kearney, embedded in the article. Thirteen sections from a first spider through item pipelines, saving to CSV and to a database, and running on a schedule in the cloud.
The closest thing on the list to the shape of your assignment: pull rows off a shop listing, put them in a dataframe, write them out. Short enough to work through in a sitting.
Goes further than the others: scrape, then clean with pandas, then chart it. The cleaning half is really week 5 arriving early, and it is the half people skip.
The only reference on the plan that covers data mining and data warehouses, which are two of the things this week promises and the other four references do not touch. Video lessons. Databases, SQL, warehouses, and what mining is for.
Sounds basic and is not. It covers the validation habits that keep a collected dataset usable, the web scraping process end to end, and Excel well enough for the collection log. Video lessons.
Start here if the five hour Scrapy course is too much for a week. HTML, requests, Beautiful Soup, and a worked example, in an hour.
Read this rather than a blog post about it. It is the crawler owner's own documentation, and it says plainly that robots.txt manages crawler traffic and is not a security or privacy mechanism, which is the point section four makes.
A fake bookstore and a quotes site, with pagination, JavaScript rendered pages, a login, a table layout and infinite scroll. Every awkward shape you will meet, on a site that exists for you to practice on.
Nothing on this list will make you write a selector. That is what the Selector Bench is for: six pages, a box, and the exact list of what came back. Read or watch one of these, then go and break something on the bench.
One collected dataset and one collection log, in the seven columns above. The dataset goes into the shared work the assignment describes; the log travels with it. If you scraped anything, say what, where, and under which terms, and attach the code.
The reference list below is the technique. This page is the judgment around it, and the live session is where the two meet.