databricks-hello-world

# Databricks Hello World ## Overview Create your first Databricks cluster and notebook to verify setup. ## Prerequisites - Completed `databricks-install-auth` setup - Valid API credentials configured - Workspace access with cluster creation permissions ## Instructions ### Step 1: Create a Cluster ```bash # Create a small development cluster via CLI databricks clusters create --json '{ "cluster_name": "hello-world-cluster", "spark_version": "14.3.x-scala2.12", "node_type_id": "Standard_DS3_v2", "autotermination_minutes": 30, "num_workers": 0, "spark_conf": { "spark.databricks.cluster.profile": "singleNode", "spark.master": "local[*]" }, "custom_tags": { "ResourceClass": "SingleNode" } }' ``` ### Step 2: Create a Notebook ```python # hello_world.py - upload as notebook from databricks.sdk import WorkspaceClient w = WorkspaceClient() # Create notebook content notebook_content = """ # Databricks Hello World # COMMAND ---------- # Simple DataFrame operations data = [("Alice", 28), ("Bob", 35), ("Charlie", 42)] df = spark.createDataFrame(data, ["name", "age"]) display(df) # COMMAND ---------- # Delta Lake example df.write.format("delta").mode("overwrite").save("/tmp/hello_world_delta") # COMMAND ---------- # Read it back df_read = spark.read.format("delta").load("/tmp/hello_world_delta") display(df_read) # COMMAND ---------- print("Hello from Databricks!") """ import base64 w.workspace.import_( path="/Users/your-email/hello_world", format="SOURCE", language="PYTHON", content=base64.b64encode(notebook_content.encode()).decode(), overwrite=True ) print("Notebook created!") ``` ### Step 3: Run the Notebook ```python from databricks.sdk import WorkspaceClient from databricks.sdk.service.jobs import Task, NotebookTask, RunNow w = WorkspaceClient() # Create a one-time run run = w.jobs.submit( run_name="hello-world-run", tasks=[ Task( task_key="hello", existing_cluster_id="your-cluster-id", notebook_task=NotebookTask( notebook_path="/Users/your-email/hello_world" ) ) ] ) # Wait for completion result = w.jobs.get_run(run.response.run_id).result() print(f"Run completed with state: {result.state.result_state}") ``` ### Step 4: Verify with CLI ```bash # List clusters databricks clusters list # Get cluster status databricks clusters get --cluster-id your-cluster-id # List workspace contents databricks workspace list /Users/your-email/ # Get run output databricks runs get-output --run-id your-run-id ``` ## Output - Development cluster created and running - Hello world notebook created in workspace - Successful notebook execution - Delta table created at `/tmp/hello_world_delta` ## Error Handling | Error | Cause | Solution | |-------|-------|----------| | `Cluster quota exceeded` | Workspace limits | Terminate unused clusters | | `Invalid node type` | Wrong instance type | Check available node types | | `Notebook path exists` | Duplicate path | Use `overwrite=True` | | `Cluster pending` | Startup in progress | Wait for `RUNNING` state | | `Permission denied` | Insufficient privileges | Request workspace admin access | ## Examples ### Interactive Cluster (Cost-Effective Dev) ```python from databricks.sdk import WorkspaceClient from databricks.sdk.service.compute import ClusterSpec w = WorkspaceClient() # Create single-node cluster for development cluster = w.clusters.create_and_wait( cluster_name="dev-cluster", spark_version="14.3.x-scala2.12", node_type_id="Standard_DS3_v2", num_workers=0, autotermination_minutes=30, spark_conf={ "spark.databricks.cluster.profile": "singleNode", "spark.master": "local[*]" } ) print(f"Cluster created: {cluster.cluster_id}") ``` ### SQL Warehouse (Serverless) ```python from databricks.sdk import WorkspaceClient w = WorkspaceClient() # Create SQL warehouse for queries warehouse = w.warehouses.create_and_wait( name="hello-warehouse", cluster_size="2X-Small", auto_stop_mins=15, warehouse_type="PRO", enable_serverless_compute=True ) print(f"Warehouse created: {warehouse.id}") ``` ### Quick DataFrame Test ```python # Run in notebook or Databricks Connect from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() # Create sample data df = spark.range(1000).toDF("id") df = df.withColumn("value", df.id * 2) # Show results df.show(5) print(f"Row count: {df.count()}") ``` ## Resources - [Databricks Quickstart](https://docs.databricks.com/getting-started/index.html) - [Cluster Configuration](https://docs.databricks.com/clusters/configure.html) - [Notebooks Guide](https://docs.databricks.com/notebooks/index.html) - [Delta Lake Quickstart](https://docs.databricks.com/delta/quick-start.html) ## Next Steps Proceed to `databricks-local-dev-loop` for local development setup.