Skip to content

First query on real data

Your first query against the samples catalog: finding a table through the catalog and schema above it, count(*) over millions of rows, and group by to see what is in a column.

Nobody has passed this one yet.

What you will learn

  • How to find a table you have never seen through the catalog and schema above it.
  • What samples is, and why a query against it gives the same answer in every workspace.
  • count(*), and why it answers as fast on a million rows as on a thousand.
  • group by, which is how you find out what is actually in a column.

The problem

The usual way to answer a question about a table is to get a copy of it. Export a CSV, open it in something, scroll, and count by eye. That works until the table is bigger than the thing you opened it in, which for real data is immediately. It is also a copy, so your answer and somebody else's answer come from two different files.

get a copy, and scroll it ? no end to it, and it is your copy or ask the table that is already there group by o_orderstatus a row per status

Set up

Nothing to set up. samples is shared into every Unity Catalog workspace by Databricks, it is read only, and samples.tpch is the TPC-H benchmark set generated to a published specification, so the answers below are the same for everybody. The starter opens on the three statements that find a table: show schemas, show tables, describe table.

Paste your salt into the first cell. One of the four answers is yours alone, so a notebook borrowed from somebody else fails that check and passes the rest.

The work

  1. Look around. List the schemas in samples, the tables in samples.tpch, and the columns of samples.tpch.orders. Every question after this is about one of those columns.
  2. Count the rows. One statement.
  3. Look inside a column. Group orders by o_orderstatus and count each group, then read two answers out of the result: how many different statuses there are, and which one is commonest.
  4. Answer for your own slice. Your salt picks a bucket out of 64. The bucket of an order is o_custkey % 64. Count the orders in yours.

Steps 2 to 4 are yours to write.

The questions

key what to print
order_rows rows in samples.tpch.orders
status_kinds how many different values o_orderstatus holds
top_status the commonest status, as the letter itself
bucket_orders orders whose customer falls in the bucket your salt picks

Notes

spark.sql(...) hands back a DataFrame, which does not run anything until you ask it for rows. display(df) draws the table, and .collect() brings it back as a list of Row you can subscript: spark.sql("select count(*) as n from ...").collect()[0]["n"].

top_status is the letter, not the count. The check compares text and ignores case.

TPC-H customer keys run from 1 upwards with no gaps, which is why o_custkey % 64 spreads orders evenly over the 64 buckets and o_orderkey would not.

We measure how the site is used. PostHog and Google Analytics, set to measurement only with advertising features off. We do not sell your data or share it for advertising, and you can turn this off. What this stores.