First query on real data
Your first query against the samples catalog: finding a table through the catalog and schema above it, count(*) over millions of rows, and group by to see what is in a column.
What you will learn
- How to find a table you have never seen through the catalog and schema above it.
- What
samplesis, and why a query against it gives the same answer in every workspace. count(*), and why it answers as fast on a million rows as on a thousand.group by, which is how you find out what is actually in a column.
The problem
The usual way to answer a question about a table is to get a copy of it. Export a CSV, open it in something, scroll, and count by eye. That works until the table is bigger than the thing you opened it in, which for real data is immediately. It is also a copy, so your answer and somebody else's answer come from two different files.
Set up
Nothing to set up. samples is shared into every Unity Catalog workspace by Databricks, it is read
only, and samples.tpch is the TPC-H benchmark set generated to a published specification, so the
answers below are the same for everybody. The starter opens on the three statements that find a
table: show schemas, show tables, describe table.
Paste your salt into the first cell. One of the four answers is yours alone, so a notebook borrowed from somebody else fails that check and passes the rest.
The work
- Look around. List the schemas in
samples, the tables insamples.tpch, and the columns ofsamples.tpch.orders. Every question after this is about one of those columns. - Count the rows. One statement.
- Look inside a column. Group
ordersbyo_orderstatusand count each group, then read two answers out of the result: how many different statuses there are, and which one is commonest. - Answer for your own slice. Your salt picks a bucket out of 64. The bucket of an order is
o_custkey % 64. Count the orders in yours.
Steps 2 to 4 are yours to write.
The questions
| key | what to print |
|---|---|
order_rows |
rows in samples.tpch.orders |
status_kinds |
how many different values o_orderstatus holds |
top_status |
the commonest status, as the letter itself |
bucket_orders |
orders whose customer falls in the bucket your salt picks |
Notes
spark.sql(...) hands back a DataFrame, which does not run anything until you ask it for rows.
display(df) draws the table, and .collect() brings it back as a list of Row you can subscript:
spark.sql("select count(*) as n from ...").collect()[0]["n"].
top_status is the letter, not the count. The check compares text and ignores case.
TPC-H customer keys run from 1 upwards with no gaps, which is why o_custkey % 64 spreads orders
evenly over the 64 buckets and o_orderkey would not.
Documentation Databricks' own pages for what this lab uses. They are reference, not the answer.
- Sample datasets
What the samples catalog holds and where it comes from.
- SHOW SCHEMAS
The statement that lists what is inside a catalog.
- SELECT
The clause order, for when a query is right but in the wrong sequence.
- GROUP BY
Grouping rows and counting each group, which is questions three and four.
- count function
What count(*) counts, and how it differs from count of a column.
-
v12026-09-21 First release.
Re-run needed means an answer changed, so passes from before it have to be earned again. Every other release leaves them alone.