<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://jkfran.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://jkfran.com/" rel="alternate" type="text/html" /><updated>2026-04-11T23:56:29+00:00</updated><id>https://jkfran.com/feed.xml</id><title type="html">Francisco Jiménez Cabrera</title><subtitle>Technical blog and free developer tools by Francisco Jiménez Cabrera. Tutorials on DevOps, Linux, Python, MLOps, and AI — plus online tools like JSON formatter, regex tester, and more.</subtitle><entry><title type="html">My Claude Code Setup: Fixing Remote Control and Running Unattended Sessions</title><link href="https://jkfran.com/claude-code-setup-remote-control-unattended-sessions/" rel="alternate" type="text/html" title="My Claude Code Setup: Fixing Remote Control and Running Unattended Sessions" /><published>2026-03-25T00:00:00+00:00</published><updated>2026-03-25T00:00:00+00:00</updated><id>https://jkfran.com/claude-code-setup-remote-control-unattended-sessions</id><content type="html" xml:base="https://jkfran.com/claude-code-setup-remote-control-unattended-sessions/"><![CDATA[<p>Claude Code’s <code class="language-plaintext highlighter-rouge">/remote-control</code> is one of those features that sounds life-changing — start a task at your desk, walk away, keep working from your phone. And it <em>is</em> life-changing, until the connection silently dies after 15–60 minutes and never recovers. The status bar shows “Remote Control reconnecting” indefinitely, and your only option is to manually cycle <code class="language-plaintext highlighter-rouge">/remote-control</code> at the terminal. Which, of course, defeats the entire purpose of remote control.</p>

<p>This is a known issue (<a href="https://github.com/anthropics/claude-code/issues/34255">anthropics/claude-code#34255</a>), and there’s a community tool called <strong>claude-remote-watchdog</strong> that auto-detects and fixes dead sessions. But getting it working properly has some gotchas nobody tells you about. Here’s the complete walkthrough, including every pitfall I hit along the way.</p>

<hr />

<h2 id="what-you-need">What You Need</h2>

<ul>
  <li><strong>Claude Code</strong> CLI (Pro or Max plan)</li>
  <li><strong>tmux</strong> — your Claude Code sessions must run inside tmux for the watchdog to see them</li>
  <li><strong>macOS or Linux</strong> (this guide uses macOS with Homebrew)</li>
</ul>

<hr />

<h2 id="step-1-install-tmux">Step 1: Install tmux</h2>

<p>If you don’t have it:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="c"># macOS</span>
brew <span class="nb">install </span>tmux

<span class="c"># Ubuntu/Debian</span>
<span class="nb">sudo </span>apt <span class="nb">install </span>tmux

<span class="c"># Fedora</span>
<span class="nb">sudo </span>dnf <span class="nb">install </span>tmux
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Verify it’s installed:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>which tmux
</pre></td></tr></tbody></table></code></pre></div></div>

<p>You should see something like <code class="language-plaintext highlighter-rouge">/opt/homebrew/bin/tmux</code>.</p>

<hr />

<h2 id="step-2-run-claude-code-inside-tmux">Step 2: Run Claude Code Inside tmux</h2>

<p>This is the critical part. The watchdog reads tmux pane content to detect stuck sessions. If Claude Code isn’t running inside tmux, the watchdog has nothing to scan.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>tmux new <span class="nt">-s</span> life
</pre></td></tr></tbody></table></code></pre></div></div>

<p>You’ll see a tmux status bar at the bottom of your terminal. Now launch Claude Code inside this session as you normally would.</p>

<p>That’s it — you’re now running Claude Code inside a tmux pane.</p>

<p><strong>Quick tmux survival guide:</strong></p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">Ctrl+B</code> then <code class="language-plaintext highlighter-rouge">D</code> — detach (session keeps running in background)</li>
  <li><code class="language-plaintext highlighter-rouge">tmux attach -t life</code> — reattach to your session</li>
  <li><code class="language-plaintext highlighter-rouge">tmux ls</code> — list sessions</li>
  <li><code class="language-plaintext highlighter-rouge">Ctrl+B</code> then <code class="language-plaintext highlighter-rouge">C</code> — new window inside the session</li>
</ul>

<hr />

<h2 id="step-3-install-the-watchdog">Step 3: Install the Watchdog</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre>git clone https://github.com/sma1lboy/claude-remote-watchdog.git
<span class="nb">cd </span>claude-remote-watchdog
./install.sh
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This creates symlinks in <code class="language-plaintext highlighter-rouge">~/.claude/</code>:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">~/.claude/commands/remote-watchdog.md</code> — slash command</li>
  <li><code class="language-plaintext highlighter-rouge">~/.claude/scripts/remote-watchdog.sh</code> — the actual watchdog script</li>
</ul>

<p>Test it manually:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>~/.claude/scripts/remote-watchdog.sh
</pre></td></tr></tbody></table></code></pre></div></div>

<p>You should see something like:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre>=== Remote Control Watchdog 22:05:42 ===
[HEALTHY] claude (%0)
[OK] All Remote Control sessions healthy
</pre></td></tr></tbody></table></code></pre></div></div>

<p>If you see <code class="language-plaintext highlighter-rouge">[SKIP] No Remote Control sessions found</code>, make sure you have <code class="language-plaintext highlighter-rouge">/remote-control</code> active inside your tmux session.</p>

<p>The install also adds a <code class="language-plaintext highlighter-rouge">/remote-watchdog</code> slash command you can run inside Claude Code. It’s nice to have for manual checks, but I went with the cronjob approach below for fully unattended recovery.</p>

<hr />

<h2 id="step-4-set-up-cron-the-right-way">Step 4: Set Up Cron (The Right Way)</h2>

<p>Use crontab to run the watchdog every 5 minutes, completely outside Claude Code:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>crontab <span class="nt">-e</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p><strong>Important:</strong> Add a <code class="language-plaintext highlighter-rouge">PATH</code> line before the cron entry. Cron runs with a minimal PATH that doesn’t include <code class="language-plaintext highlighter-rouge">/opt/homebrew/bin</code>, so it can’t find <code class="language-plaintext highlighter-rouge">tmux</code>. Without this, the watchdog will run but report <code class="language-plaintext highlighter-rouge">[SKIP]</code> every single time because it literally cannot execute the <code class="language-plaintext highlighter-rouge">tmux</code> commands it needs.</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>PATH=/opt/homebrew/bin:/usr/bin:/bin
*/5 * * * * ~/.claude/scripts/remote-watchdog.sh &gt;&gt; /tmp/remote-watchdog.log 2&gt;&amp;1
</pre></td></tr></tbody></table></code></pre></div></div>

<p>If you’ve never used crontab before, you’ll see:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>crontab: no crontab for yourname - using an empty one
crontab: installing new crontab
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That’s normal. It created a fresh crontab with your entry.</p>

<hr />

<h2 id="step-5-disable-the-update-banner">Step 5: Disable the Update Banner</h2>

<p>This is the second gotcha that took a while to figure out. Claude Code shows an “Update available! Run: brew upgrade claude-code” banner in the status area — the same area where the Remote Control status normally appears. When this banner is showing, the watchdog captures that text instead of the Remote Control status, so it reports <code class="language-plaintext highlighter-rouge">[SKIP]</code> even though your session is right there.</p>

<p>The fix is to set <code class="language-plaintext highlighter-rouge">DISABLE_AUTOUPDATER=1</code>. More on where to put this below.</p>

<hr />

<h2 id="step-6-set-up-your-shell-alias">Step 6: Set Up Your Shell Alias</h2>

<p>By now you probably want a quick command that launches Claude Code with all the right settings. Add this to your <code class="language-plaintext highlighter-rouge">~/.zshrc</code>:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>c<span class="o">()</span> <span class="o">{</span> <span class="nv">DISABLE_AUTOUPDATER</span><span class="o">=</span>1 claude <span class="nt">--effort</span> max <span class="nt">--dangerously-skip-permissions</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span> <span class="o">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Then reload:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="nb">source</span> ~/.zshrc
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Now <code class="language-plaintext highlighter-rouge">c</code> gives you:</p>

<ul>
  <li><strong><code class="language-plaintext highlighter-rouge">DISABLE_AUTOUPDATER=1</code></strong> — kills the update banner so the watchdog can see the status bar</li>
  <li><strong><code class="language-plaintext highlighter-rouge">--effort max</code></strong> — maximum reasoning depth (Opus only)</li>
  <li><strong><code class="language-plaintext highlighter-rouge">--dangerously-skip-permissions</code></strong> — no permission prompts for every command</li>
  <li><strong><code class="language-plaintext highlighter-rouge">"$@"</code></strong> — passes through any extra arguments, so <code class="language-plaintext highlighter-rouge">c -c</code> continues your last session</li>
</ul>

<hr />

<h2 id="step-7-skip-the-safety-confirmation">Step 7: Skip the Safety Confirmation</h2>

<p>When you run <code class="language-plaintext highlighter-rouge">--dangerously-skip-permissions</code>, Claude Code shows a scary warning and asks you to confirm. To skip it, add this to <code class="language-plaintext highlighter-rouge">~/.claude/settings.json</code>:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"skipDangerousModePermissionPrompt"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>

<p>If you already have settings in that file, just add the key to the existing JSON.</p>

<hr />

<h2 id="step-8-enable-remote-control-by-default">Step 8: Enable Remote Control by Default</h2>

<p>So you don’t have to type <code class="language-plaintext highlighter-rouge">/remote-control</code> every time you start a session, run <code class="language-plaintext highlighter-rouge">/config</code> inside Claude Code and set <strong>“Enable Remote Control for all sessions”</strong> to true. Every new Claude Code session will automatically start with Remote Control active.</p>

<hr />

<h2 id="checking-the-logs">Checking the Logs</h2>

<p>Your watchdog logs to <code class="language-plaintext highlighter-rouge">/tmp/remote-watchdog.log</code>. Check it anytime:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="c"># Last few entries</span>
<span class="nb">tail</span> <span class="nt">-20</span> /tmp/remote-watchdog.log

<span class="c"># Follow live</span>
<span class="nb">tail</span> <span class="nt">-f</span> /tmp/remote-watchdog.log
</pre></td></tr></tbody></table></code></pre></div></div>

<p>A healthy log looks like:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>=== Remote Control Watchdog 13:40:00 ===
[WARN] claude (%1): 'reconnecting' — confirming next check
=== Remote Control Watchdog 13:45:00 ===
[DEAD] claude (%1): stuck on 'reconnecting' — auto-reconnecting
[ACTION] Cycling /remote-control on pane %1 (claude)...
[OK] Reconnect sequence sent to pane %1 (claude)
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The watchdog uses a 2-check grace period — <code class="language-plaintext highlighter-rouge">[WARN]</code> on first detection, <code class="language-plaintext highlighter-rouge">[DEAD]</code> and auto-reconnect on the second. This avoids false positives on transient drops.</p>

<hr />

<h2 id="how-the-fix-actually-works">How the Fix Actually Works</h2>

<p>When the watchdog detects a stuck session, it sends tmux keystrokes to cycle the <code class="language-plaintext highlighter-rouge">/remote-control</code> menu:</p>

<ol>
  <li><code class="language-plaintext highlighter-rouge">Ctrl+C</code> to clear the prompt</li>
  <li>Types <code class="language-plaintext highlighter-rouge">/remote-control</code> which opens the TUI menu</li>
  <li>Navigates to “Disconnect this session” and presses Enter</li>
  <li>Types <code class="language-plaintext highlighter-rouge">/remote-control</code> again, which auto-connects to a fresh bridge</li>
</ol>

<p>Your actual Claude Code session — with its full conversation history — never stops. Only the remote bridge gets cycled. Think of it like unplugging and re-plugging a cable.</p>

<hr />

<h2 id="known-issue-effort-level-max-doesnt-persist">Known Issue: Effort Level “max” Doesn’t Persist</h2>

<p>If you set <code class="language-plaintext highlighter-rouge">"effortLevel": "max"</code> in <code class="language-plaintext highlighter-rouge">settings.json</code>, it gets silently downgraded to “high” when you interact with the <code class="language-plaintext highlighter-rouge">/model</code> UI. This is a known bug. The workaround is using the <code class="language-plaintext highlighter-rouge">--effort max</code> CLI flag every time, which is why we put it in the shell alias above.</p>

<hr />

<h2 id="the-complete-setup-checklist">The Complete Setup Checklist</h2>

<ol>
  <li>Install tmux (<code class="language-plaintext highlighter-rouge">brew install tmux</code>)</li>
  <li>Start Claude Code inside tmux (<code class="language-plaintext highlighter-rouge">tmux new -s life</code>, then launch Claude)</li>
  <li>Install the watchdog (<code class="language-plaintext highlighter-rouge">git clone</code> + <code class="language-plaintext highlighter-rouge">./install.sh</code>)</li>
  <li>Set up crontab with the correct PATH</li>
  <li>Disable the update banner (<code class="language-plaintext highlighter-rouge">DISABLE_AUTOUPDATER=1</code>)</li>
  <li>Add the shell alias to <code class="language-plaintext highlighter-rouge">~/.zshrc</code></li>
  <li>Skip the safety confirmation in <code class="language-plaintext highlighter-rouge">~/.claude/settings.json</code></li>
  <li>Enable Remote Control for all sessions via <code class="language-plaintext highlighter-rouge">/config</code></li>
</ol>

<hr />

<h2 id="quick-reference">Quick Reference</h2>

<table>
  <thead>
    <tr>
      <th>What</th>
      <th>Command</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Start Claude with everything</td>
      <td><code class="language-plaintext highlighter-rouge">c</code> (or <code class="language-plaintext highlighter-rouge">c -c</code> to continue)</td>
    </tr>
    <tr>
      <td>Check watchdog logs</td>
      <td><code class="language-plaintext highlighter-rouge">tail -20 /tmp/remote-watchdog.log</code></td>
    </tr>
    <tr>
      <td>Run watchdog manually</td>
      <td><code class="language-plaintext highlighter-rouge">~/.claude/scripts/remote-watchdog.sh</code></td>
    </tr>
    <tr>
      <td>Check crontab</td>
      <td><code class="language-plaintext highlighter-rouge">crontab -l</code></td>
    </tr>
    <tr>
      <td>Check tmux sessions</td>
      <td><code class="language-plaintext highlighter-rouge">tmux ls</code></td>
    </tr>
    <tr>
      <td>Attach to tmux</td>
      <td><code class="language-plaintext highlighter-rouge">tmux attach -t life</code></td>
    </tr>
    <tr>
      <td>Detach from tmux</td>
      <td><code class="language-plaintext highlighter-rouge">Ctrl+B</code> then <code class="language-plaintext highlighter-rouge">D</code></td>
    </tr>
  </tbody>
</table>

<hr />

<p>Remote Control is genuinely useful when it works — start a task on your laptop, continue from your phone on the couch. The connection bug makes it unreliable, but with this watchdog setup, the dead sessions get automatically revived without you lifting a finger. Set it up once and forget about it.</p>]]></content><author><name>jkfran</name></author><category term="tools" /><summary type="html"><![CDATA[A complete guide to fixing Claude Code remote control disconnections using tmux, the claude-remote-watchdog, and cron.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/claude-remote-control.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/claude-remote-control.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">QdrantSync: Simplifying Data Migration Between Qdrant Instances</title><link href="https://jkfran.com/qdrantsync-simplifying-data-migration/" rel="alternate" type="text/html" title="QdrantSync: Simplifying Data Migration Between Qdrant Instances" /><published>2025-01-24T00:00:00+00:00</published><updated>2025-01-24T00:00:00+00:00</updated><id>https://jkfran.com/qdrantsync-simplifying-data-migration</id><content type="html" xml:base="https://jkfran.com/qdrantsync-simplifying-data-migration/"><![CDATA[<p>I recently developed <strong>QdrantSync</strong>, a CLI tool to simplify and streamline migrating collections and data points between <a href="https://qdrant.tech/documentation/">Qdrant</a> instances. It was born out of my experience with Qdrant snapshots, which can be tedious and complex—especially when moving data to clusters with different configurations or sizes.</p>

<h2 id="why-qdrantsync">Why QdrantSync?</h2>

<p>Snapshots are powerful, but they’re not always the best option for every scenario. Challenges arise when migrating data:</p>

<ul>
  <li><strong>Cluster Size Differences</strong>: Snapshots assume identical setups, making it tricky to adjust for varying cluster configurations.</li>
  <li><strong>Flexibility</strong>: Adapting data, schema, or replication factors during migration requires extra effort.</li>
  <li><strong>Incremental Updates</strong>: Snapshots don’t support partial or staged migrations easily.</li>
</ul>

<p>QdrantSync solves these pain points by providing a robust and flexible alternative for seamless data transfer.</p>

<h3 id="key-features">Key Features</h3>

<ul>
  <li><strong>Customizable Migration</strong>: Fine-tune schema settings like replication factors and prefixes to suit your destination cluster.</li>
  <li><strong>Incremental Migration</strong>: Mark and track migrated data, allowing you to resume or refresh migrations without duplication.</li>
  <li><strong>Scalable Batch Processing</strong>: Scroll through large datasets efficiently with real-time progress tracking via <code class="language-plaintext highlighter-rouge">tqdm</code>.</li>
  <li><strong>Error Handling</strong>: Safe operations ensure no unintended overwrites or data loss, with options to continue migrations for existing collections.</li>
</ul>

<h3 id="getting-started">Getting Started</h3>

<ol>
  <li>
    <p><strong>Install</strong>:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>pip <span class="nb">install </span>QdrantSync
qdrantsync <span class="nt">--help</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p><strong>Migrate Data</strong>:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>qdrantsync <span class="nt">--source-url</span> &lt;<span class="nb">source</span><span class="o">&gt;</span> <span class="nt">--destination-url</span> &lt;destination&gt; <span class="nt">--migration-id</span> &lt;<span class="nb">id</span><span class="o">&gt;</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
</ol>

<h3 id="use-cases">Use Cases</h3>

<ul>
  <li>Migrate Qdrant data between environments (e.g., staging to production).</li>
  <li>Upgrade infrastructure or move to a different cloud provider.</li>
  <li>Perform selective or incremental backups.</li>
</ul>

<h3 id="contribute">Contribute</h3>

<p>I’d love to hear your feedback or see contributions! The project is open-source and MIT-licensed.</p>

<h3 id="github-repo">GitHub Repo</h3>

<p>Check it out here: <a href="https://github.com/jkfran/QdrantSync">GitHub</a>.</p>

<p>Have you run into similar challenges with Qdrant snapshots? Let me know your thoughts or suggestions!</p>]]></content><author><name>jkfran</name></author><category term="devops" /><summary type="html"><![CDATA[Introducing QdrantSync, a CLI tool for migrating collections and data points between Qdrant vector database instances.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/self-hosted-vector-database.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/self-hosted-vector-database.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Validating JSON Outputs in MLflow with Custom Metrics</title><link href="https://jkfran.com/mlflow-custom-metrics-json/" rel="alternate" type="text/html" title="Validating JSON Outputs in MLflow with Custom Metrics" /><published>2024-03-21T00:00:00+00:00</published><updated>2024-03-21T00:00:00+00:00</updated><id>https://jkfran.com/mlflow-custom-metrics-json</id><content type="html" xml:base="https://jkfran.com/mlflow-custom-metrics-json/"><![CDATA[<p>Welcome to a new post on my blog. We’ll delve into an interesting aspect of working with MLflow – adding a custom metric to validate JSON outputs. This tutorial is particularly useful for developers and data scientists looking to ensure the integrity and structure of their model’s outputs when JSON is expected.</p>

<h2 id="why-validate-json-outputs">Why Validate JSON Outputs?</h2>

<p>In the world of machine learning and data science, models often need to output data in structured formats, JSON being one of the most popular due to its versatility and wide adoption in web services and applications. Ensuring your model reliably produces valid JSON responses is crucial, especially in production environments where data consistency and integrity are paramount.</p>

<h3 id="introducing-custom-metrics-in-mlflow">Introducing Custom Metrics in MLflow</h3>

<p>MLflow, an open-source platform for the machine learning lifecycle, includes capabilities for tracking experiments, packaging code into reproducible runs, and managing models. However, it might not natively support specific validation checks like verifying JSON output. This is where custom metrics come into play.</p>

<h3 id="step-by-step-guide-to-creating-a-json-validity-metric">Step-by-Step Guide to Creating a JSON Validity Metric</h3>

<p>Below is a detailed walkthrough on how to implement a custom metric in MLflow for checking JSON validity. This script demonstrates adding such a metric and using it to evaluate a model’s predictions.</p>

<h4 id="1-define-the-json-validity-evaluation-function">1. Define the JSON Validity Evaluation Function</h4>

<p>First, we define an evaluation function that checks if a given string is valid JSON. This function iterates through each model prediction, validating each one and appending the result (1 for valid, 0 for invalid) to a list of scores.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">_json_validity_eval_fn</span><span class="p">(</span><span class="n">outputs</span><span class="p">,</span> <span class="n">references</span><span class="p">):</span>
    <span class="n">validity_scores</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">.</span><span class="n">iterrows</span><span class="p">():</span>
        <span class="n">prediction</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s">"prediction"</span><span class="p">]</span>
        <span class="k">if</span> <span class="n">_is_valid_json</span><span class="p">(</span><span class="n">prediction</span><span class="p">):</span>
            <span class="n">validity_scores</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">validity_scores</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">MetricValue</span><span class="p">(</span><span class="n">validity_scores</span><span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h4 id="2-implement-a-helper-function-to-check-json-validity">2. Implement a Helper Function to Check JSON Validity</h4>

<p>A helper function uses Python’s <code class="language-plaintext highlighter-rouge">json.loads</code> method to determine if a string is a valid JSON. It returns True for valid JSON strings and False otherwise.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">_is_valid_json</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">True</span>
    <span class="k">except</span> <span class="nb">ValueError</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">False</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h4 id="3-create-the-custom-metric">3. Create the Custom Metric</h4>

<p>We then wrap our evaluation function in a custom metric definition using MLflow’s <code class="language-plaintext highlighter-rouge">make_metric</code> function, specifying our evaluation function, whether a higher score is better, and a name for the metric.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">json_validity</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">EvaluationMetric</span><span class="p">:</span>
    <span class="k">return</span> <span class="n">make_metric</span><span class="p">(</span>
        <span class="n">eval_fn</span><span class="o">=</span><span class="n">_json_validity_eval_fn</span><span class="p">,</span>
        <span class="n">greater_is_better</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
        <span class="n">name</span><span class="o">=</span><span class="s">"json_validity"</span><span class="p">,</span>
    <span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h4 id="4-evaluate-the-model">4. Evaluate the Model</h4>

<p>With the custom metric defined, we can now use it to evaluate a model’s output. In this example, we use a remote tracking server plus MLflow deployments, feel free to adapt this to your needs, the DataFrame in my example are two inputs designed to produce JSON outputs and invoke <code class="language-plaintext highlighter-rouge">mlflow.evaluate</code> with our custom metric.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="c1"># Point the client to the local MLflow Deployments Server and set tracking URI
</span><span class="n">set_deployments_target</span><span class="p">(</span><span class="s">"http://localhost:7000"</span><span class="p">)</span>
<span class="n">mlflow</span><span class="p">.</span><span class="n">set_tracking_uri</span><span class="p">(</span><span class="s">"http://localhost:5000"</span><span class="p">)</span>

<span class="c1"># Evaluate the model with the custom metric
</span><span class="k">with</span> <span class="n">mlflow</span><span class="p">.</span><span class="n">start_run</span><span class="p">()</span> <span class="k">as</span> <span class="n">run</span><span class="p">:</span>
    <span class="n">results</span> <span class="o">=</span> <span class="n">mlflow</span><span class="p">.</span><span class="n">evaluate</span><span class="p">(</span>
        <span class="n">model</span><span class="o">=</span><span class="s">"endpoints:/chatgpt-35-turbo"</span><span class="p">,</span>
        <span class="n">data</span><span class="o">=</span><span class="n">eval_data</span><span class="p">,</span>
        <span class="n">inference_params</span><span class="o">=</span><span class="p">{</span><span class="s">"max_tokens"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span> <span class="s">"temperature"</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">},</span>
        <span class="n">extra_metrics</span><span class="o">=</span><span class="p">[</span><span class="n">json_validity</span><span class="p">()],</span>
    <span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h4 id="final-code">Final code</h4>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
</pre></td><td class="rouge-code"><pre><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">mlflow</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="kn">from</span> <span class="nn">mlflow.deployments</span> <span class="kn">import</span> <span class="n">set_deployments_target</span>
<span class="kn">from</span> <span class="nn">mlflow.metrics.base</span> <span class="kn">import</span> <span class="n">MetricValue</span>
<span class="kn">from</span> <span class="nn">mlflow.models</span> <span class="kn">import</span> <span class="n">EvaluationMetric</span><span class="p">,</span> <span class="n">make_metric</span>


<span class="k">def</span> <span class="nf">_json_validity_eval_fn</span><span class="p">(</span><span class="n">outputs</span><span class="p">,</span> <span class="n">references</span><span class="p">):</span>
    <span class="c1"># Initialize a list to store validity scores
</span>    <span class="n">validity_scores</span> <span class="o">=</span> <span class="p">[]</span>

    <span class="c1"># Iterate over each row in the DataFrame
</span>    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">.</span><span class="n">iterrows</span><span class="p">():</span>
        <span class="c1"># Extract the prediction from the current row
</span>        <span class="n">prediction</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s">"prediction"</span><span class="p">]</span>

        <span class="c1"># Check if the prediction is a valid JSON
</span>        <span class="k">if</span> <span class="n">_is_valid_json</span><span class="p">(</span><span class="n">prediction</span><span class="p">):</span>
            <span class="n">validity_scores</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>  <span class="c1"># Valid JSON
</span>        <span class="k">else</span><span class="p">:</span>
            <span class="n">validity_scores</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>  <span class="c1"># Invalid JSON
</span>
    <span class="c1"># Return a MetricValue object with the scores
</span>    <span class="k">return</span> <span class="n">MetricValue</span><span class="p">(</span><span class="n">validity_scores</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">_is_valid_json</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="s">"""
    Helper function to check if a string is a valid JSON.
    """</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">True</span>
    <span class="c1"># json.decoder.JSONDecodeError inherits from ValueError
</span>    <span class="k">except</span> <span class="nb">ValueError</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">False</span>


<span class="k">def</span> <span class="nf">json_validity</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">EvaluationMetric</span><span class="p">:</span>
    <span class="s">"""
    Creates a metric for evaluating the validity of JSON strings produced by a model.
    """</span>
    <span class="k">return</span> <span class="n">make_metric</span><span class="p">(</span>
        <span class="n">eval_fn</span><span class="o">=</span><span class="n">_json_validity_eval_fn</span><span class="p">,</span>
        <span class="n">greater_is_better</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
        <span class="n">name</span><span class="o">=</span><span class="s">"json_validity"</span><span class="p">,</span>
    <span class="p">)</span>


<span class="c1"># Point the client to the local MLflow Deployments Server
</span><span class="n">set_deployments_target</span><span class="p">(</span><span class="s">"http://localhost:7000"</span><span class="p">)</span>
<span class="n">mlflow</span><span class="p">.</span><span class="n">set_tracking_uri</span><span class="p">(</span><span class="s">"http://localhost:5000"</span><span class="p">)</span>

<span class="c1"># Create a test case of inputs that will be passed into the model and ground_truth
</span><span class="n">eval_data</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span>
    <span class="p">{</span>
        <span class="s">"inputs"</span><span class="p">:</span> <span class="p">[</span>
            <span class="s">'Convert the following description into a JSON object: MLflow is an open source platform for the machine learning lifecycle, including experimentation, reproducibility, and deployment. Structure the JSON with keys for "name", "description", and "features".'</span><span class="p">,</span>
            <span class="s">"Provide a brief explanation of what Apache Spark is."</span><span class="p">,</span>
        <span class="p">]</span>
    <span class="p">}</span>
<span class="p">)</span>

<span class="k">with</span> <span class="n">mlflow</span><span class="p">.</span><span class="n">start_run</span><span class="p">()</span> <span class="k">as</span> <span class="n">run</span><span class="p">:</span>
    <span class="n">results</span> <span class="o">=</span> <span class="n">mlflow</span><span class="p">.</span><span class="n">evaluate</span><span class="p">(</span>
        <span class="n">model</span><span class="o">=</span><span class="s">"endpoints:/chatgpt-35-turbo"</span><span class="p">,</span>
        <span class="n">data</span><span class="o">=</span><span class="n">eval_data</span><span class="p">,</span>
        <span class="n">inference_params</span><span class="o">=</span><span class="p">{</span><span class="s">"max_tokens"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span> <span class="s">"temperature"</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">},</span>
        <span class="c1"># model_type="question-answering",
</span>        <span class="c1"># Include the custom metric for JSON validation
</span>        <span class="n">extra_metrics</span><span class="o">=</span><span class="p">[</span><span class="n">json_validity</span><span class="p">()],</span>
    <span class="p">)</span>

    <span class="c1"># Print aggregated evaluation results
</span>    <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Aggregated evaluation results: </span><span class="se">\n</span><span class="si">{</span><span class="n">results</span><span class="p">.</span><span class="n">metrics</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>

    <span class="c1"># Evaluation result for each data record is available in results.tables
</span>    <span class="n">eval_table</span> <span class="o">=</span> <span class="n">results</span><span class="p">.</span><span class="n">tables</span><span class="p">[</span><span class="s">"eval_results_table"</span><span class="p">]</span>
    <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Evaluation table: </span><span class="se">\n</span><span class="si">{</span><span class="n">eval_table</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="conclusion">Conclusion</h3>

<p>Adding custom metrics to MLflow allows for flexible and precise evaluation of your models, tailored to your specific needs. By validating JSON outputs, you ensure that your model meets the requirements for structured data output, enhancing its reliability and applicability in real-world scenarios.</p>

<p>I hope this tutorial has been helpful. As always, I encourage you to experiment with this script and adapt it to your projects. Happy coding!</p>]]></content><author><name>jkfran</name></author><category term="python" /><summary type="html"><![CDATA[How to create custom metrics in MLflow to validate JSON outputs from machine learning models.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/mlflow-custom-metrics.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/mlflow-custom-metrics.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Selecting the Ideal Self-Hosted Vector Database</title><link href="https://jkfran.com/selecting-ideal-self-hosted-vector-database/" rel="alternate" type="text/html" title="Selecting the Ideal Self-Hosted Vector Database" /><published>2023-05-15T00:00:00+00:00</published><updated>2023-05-15T00:00:00+00:00</updated><id>https://jkfran.com/selecting-ideal-self-hosted-vector-database</id><content type="html" xml:base="https://jkfran.com/selecting-ideal-self-hosted-vector-database/"><![CDATA[<p>As an MLOps engineer, I was recently entrusted with the responsibility of choosing the most suitable Vector Database to address one of our crucial Data Science needs. In this case, it was the need for seamless integration with the widely-used <a href="https://python.langchain.com/en/latest/index.html">Langchain</a> library. My task was to make the selection from the MLOps perspective, aiming to identify a self-hosted vector database that would meet the requirements.
After a preliminary selection from the most popular vector/embedding databases, the potential candidates are Milvus, Pinecone, Qdrant, and PGVector (Postgres). With these options at hand, I had the opportunity to evaluate each database in terms of:</p>

<ul>
  <li>Scale</li>
  <li>Performance</li>
  <li>Disaster recovery</li>
</ul>

<p>In this blog post, I’ll walk you through my research process, offering insights into the various aspects I considered and revealing why we eventually decided on Qdrant. Let’s dive into the journey of this decision-making process.</p>

<hr />

<h2 id="milvus-a-renowned-name-with-robust-architecture">Milvus: A Renowned Name with Robust Architecture</h2>

<p>The first port of call on our journey was Milvus, an esteemed entity in the realm of vector databases. With a multi-layered, robust architecture, it has gained significant traction on GitHub, cementing its position in the landscape of popular vector databases.</p>

<p>At its core, Milvus boasts a design that is both comprehensive and sophisticated. Its default configuration deploys a considerable number of pods, reflecting an impressive level of scalability and resilience. However, this also means that it demands substantial resources, which, while suitable for some, proved to be a bit too resource-intensive for our specific needs.</p>

<p>Despite the undeniable merits of Milvus, including its high performance, scalable architecture, and strong community support, it felt like a larger tool than we required for our particular scenario. As a result, its advanced functionality and the operational overhead associated with managing such a system seemed somewhat disproportionate to our use case.</p>

<p>While Milvus undoubtedly excels in many dimensions, it underscored the importance of aligning the capabilities of a tool with our project’s specific needs, a lesson we carried forward in our quest for the optimal vector database.</p>

<h2 id="pinecone-a-powerful-proprietary-solution">Pinecone: A Powerful Proprietary Solution</h2>

<p>Our exploration then led us to Pinecone, a fully managed vector database renowned for adeptly handling unstructured search engine requirements. Pinecone distinguishes itself with its intuitive features and streamlined operations, which were evident in the recent 2.0 release.</p>

<p>The standout feature in this new release was the introduction of single-stage filtering. This innovation greatly simplifies data querying, allowing users to retrieve relevant data more efficiently, without the need for multiple filtering stages. This unique aspect undoubtedly adds value, especially for teams seeking streamlined and efficient data management.</p>

<p>However, despite Pinecone scoring highly on most of our key considerations – such as performance, scale, and data persistence – it fell short in a couple of critical areas for us. Firstly, Pinecone is a proprietary paid solution and not an open-source platform. Secondly, Pinecone does not provide a self-hosted option. This was a crucial requirement for us, as we were specifically seeking a self-hosted vector database to maintain greater control over our data and operations.</p>

<p>In conclusion, while Pinecone’s impressive capabilities and innovative features make it an excellent choice for many, it was not the perfect fit for our specific scenario due to its proprietary nature and lack of a self-hosting option. But, I can see Pinecone as the perfect choice for companies that need a solution but don’t want to deal with self-hosting this kind of service.</p>

<h2 id="qdrant-a-robust-rust-built-vector-database">Qdrant: A Robust Rust-built Vector Database</h2>

<p>The next milestone on our exploration was Qdrant, a vector database built entirely in Rust. As we delved deeper into our research, it quickly became evident that Qdrant was a formidable contender in the arena of vector databases.</p>

<p>One of the key aspects that set Qdrant apart was its dynamic query planning. This feature allows for more efficient processing of queries, resulting in quicker retrieval of relevant information. The payload data indexing feature also emerged as a major highlight, contributing to faster data access and improved search capabilities.</p>

<p>Another standout element was Qdrant’s Scalar Quantization feature. Often mentioned in discussions and articles, this feature is noted for its significant role in enhancing performance and efficiency. It achieves this by reducing the size of stored vectors while maintaining their distinct characteristics, leading to optimized resource utilization.</p>

<p>A major attraction of Qdrant was its ease to run container. This allows for smooth deployment and management of Qdrant within a Kubernetes environment, which is particularly beneficial for teams using container orchestration systems.</p>

<p>Despite the many strong points of Qdrant, our research did uncover some online concerns like missing <a href="https://github.com/qdrant/qdrant/issues/1739">authentication in the Qdrant API</a>, this feature was addressed <a href="https://github.com/qdrant/qdrant/pull/1745">recently</a>. But, it is in the development branch. However, this is a minor issue for us since we are not exposing the database to the outside. This is a relatively new Database and offers excellent performance, making it an enticing choice for our needs.</p>

<p>In the end, the combination of dynamic query planning, payload data indexing, Scalar Quantization, and seamless Kubernetes integration swayed us in Qdrant’s favor. Despite minor concerns, its robust performance, efficiency, and compatibility made it an ideal choice for our specific requirements.</p>

<h2 id="pgvector-a-trusted-postgres-extension-with-scaling-challenges">PGVector: A Trusted Postgres Extension with Scaling Challenges</h2>

<p>Finally, we turned our attention to PGVector, an extension of the widely trusted PostgreSQL database. PostgreSQL’s reputation as a robust and reliable solution for many businesses initially made PGVector an intriguing option in our quest for the ideal vector database.</p>

<p>However, upon further research, a few limitations came to light. One significant concern was that scaling PGVector within a Kubernetes cluster could pose challenges. Kubernetes is often used for managing containerized workloads and services, and any difficulties in scaling within this environment could hinder operational efficiency.</p>

<p>Another aspect where PGVector seemed to falter was performance. Compared to its competitors, search operations in PGVector were reported to be slower. While this might not be an issue for smaller scale projects, it could potentially become a bottleneck in more demanding scenarios, affecting the overall efficiency of data retrieval.</p>

<p>On a positive note, with Postgres there are plenty of tools and integrations already available online, a key factor in our evaluation was disaster recovery, and Postgres definetly scores high in this category. However, the lower performance and scalability scores made it less appealing compared to the other options we were considering.</p>

<p>In conclusion, despite its roots in the well-regarded PostgreSQL database, the challenges related to scaling and performance led us to explore other options.</p>

<hr />

<h2 id="the-final-choice-qdrant---a-winning-combination-of-performance-and-scalability">The Final Choice: Qdrant - A Winning Combination of Performance and Scalability</h2>

<p>After conducting comprehensive research and carefully evaluating our options, we found Qdrant to be the ideal choice for our specific needs. This Rust-built vector database showcased superior performance and exhibited a significant edge over its competitors in a number of key areas.</p>

<p>Qdrant’s seamless scalability within a Kubernetes environment was a major factor in our decision, as it ensures the database can grow and adapt to our evolving needs. Moreover, standout features like dynamic query planning and payload data indexing further solidified its position as our top choice. These features collectively contribute to efficient data retrieval and improved search capabilities, which are critical to our operations.</p>

<p>For those setting off on a similar journey in the world of vector databases, we plan to use the <a href="https://github.com/qdrant/qdrant-helm/">official Helm chart</a> for deploying Qdrant in our Kubernetes clusters. This resource provides a reliable, streamlined approach to deployment, simplifying the integration process.</p>

<p>In conclusion, this exploration through the landscape of vector databases has been a valuable and enlightening experience. I’m eager to see how Qdrant’s robust capabilities will enhance our operations at Builder.ai. I hope that sharing our journey will provide useful insights for others navigating the complexities of vector database selection. Until next time, happy coding!</p>]]></content><author><name>jkfran</name></author><category term="ai" /><summary type="html"><![CDATA[Comparing self-hosted vector databases — Milvus, Pinecone, Qdrant, and PGVector — from an MLOps perspective.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/self-hosted-vector-database.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/self-hosted-vector-database.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Introduction to vector / embedding databases</title><link href="https://jkfran.com/introduction-vector-embedding-databases/" rel="alternate" type="text/html" title="Introduction to vector / embedding databases" /><published>2023-05-12T00:00:00+00:00</published><updated>2023-05-12T00:00:00+00:00</updated><id>https://jkfran.com/introduction-vector-embedding-databases</id><content type="html" xml:base="https://jkfran.com/introduction-vector-embedding-databases/"><![CDATA[<p>In recent years, there has been a growing interest in vector/embedding databases. These databases are designed to store and query vector representations of data, such as text, images, and audio. Vector representations are powerful tools for representing the meaning of data, and they can be used for a variety of tasks, such as search, recommendation, and machine learning.</p>

<p>In this blog post, we will provide an introduction to vector databases. We will discuss the different types of vector representations, and we will explain how vector databases work. We will also discuss some of the benefits of using vector databases.</p>

<h2 id="what-are-vector-representations">What are Vector Representations?</h2>

<p>Vector representations serve as a bridge, translating various forms of data - such as text, images, and audio - into numeric form. This conversion facilitates the encapsulation of meaning, characteristics, or features from the original data, making it more accessible for computational processing and analysis.</p>

<p>Several types of vector representations are commonly used, each with its unique advantages and applications. Here’s a closer look at some of these:</p>

<ul>
  <li>
    <p><strong>Word Embeddings:</strong> These are vector representations specifically designed for words, capturing their semantic meanings. These embeddings are often derived from extensive text corpora, leveraging machine learning techniques to represent the nuanced relationships between words. Word embeddings find wide-ranging applications across numerous tasks, including text classification, machine translation, and question answering.</p>
  </li>
  <li>
    <p><strong>Image Embeddings:</strong> Just as words can be translated into numeric form, images can be transformed into vector representations as well. Image embeddings distill visual content into a format that machines can understand and process. These embeddings, generally learned from large image datasets, can represent the content of images, enabling tasks like image classification, image retrieval, and object detection.</p>
  </li>
  <li>
    <p><strong>Audio Embeddings:</strong> For audio data, audio embeddings provide a means to capture and represent the distinct characteristics of sound. These vector representations, trained on extensive audio corpora, can encapsulate the content of audio recordings. Applications for audio embeddings are diverse and include speech recognition, speaker identification, and music genre classification.</p>
  </li>
</ul>

<p><img src="https://github.com/jkfran/jkfran.com/assets/6353928/3176a677-479d-4f5d-b3b2-a3bb721da1e2" alt="Vector embedding similarity diagram" width="1136" height="535" loading="lazy" /></p>

<p>In essence, vector representations provide a powerful tool to translate various forms of data into a language that machines can understand, process, and learn from, paving the way for a broad spectrum of data analysis and machine learning tasks.</p>

<h2 id="how-do-vector-databases-work">How do Vector Databases Work?</h2>

<p>At their core, vector databases are engineered to handle the storage and querying of vector representations of data. They employ a mix of strategies to efficiently manage these vector representations, making it possible to retrieve relevant data quickly and accurately. Let’s delve into some of these techniques:</p>

<ul>
  <li>
    <p><strong>Hierarchical Indexes:</strong> Imagine storing and organizing data in a tree-like structure, where each branch leads you closer to the information you’re seeking. That’s essentially what hierarchical indexing does. It allows vector databases to swiftly locate vector representations similar to a given vector, reducing search times and increasing efficiency.</p>
  </li>
  <li>
    <p><strong>Spatial Indexes:</strong> Spatial indexing involves using specific data structures, like kd-trees or quadtrees, which are designed to handle multi-dimensional data. This approach allows vector databases to rapidly find vector representations that are in close proximity to a given vector. In other words, it’s like having a map that guides you to the data points that are ‘nearest’ to your location in a multi-dimensional space.</p>
  </li>
  <li>
    <p><strong>Graph Indexes:</strong> Graph indexing makes use of graph data structures to store and query vector representations. If you picture your data as a network of interconnected points, then graph indexing helps you find the data points that are directly linked to a given vector. It’s akin to finding friends-of-friends in a social network.</p>
  </li>
</ul>

<p>In a nutshell, vector databases utilize these techniques to efficiently navigate the high-dimensional space of vector representations, making it possible to quickly retrieve the data that’s most relevant to your query. This functionality is integral to many machine learning applications and data analysis tasks.</p>

<h2 id="benefits-of-using-vector-databases">Benefits of Using Vector Databases</h2>

<p>There are many benefits to using vector databases. Some of the benefits of using vector databases include:</p>

<ul>
  <li><strong>Faster Search:</strong> Vector databases can be used to quickly find vector representations that are similar to a given vector representation. This can be used to improve the performance of search applications, such as search engines and recommender systems.</li>
  <li><strong>More Accurate Search:</strong> Vector databases can be used to find vector representations that are more semantically similar to a given vector representation. This can be used to improve the accuracy of search applications, such as search engines and recommender systems.</li>
  <li><strong>More Flexible Search:</strong> Vector databases can be used to perform a variety of search queries, such as nearest neighbor search, range search, and keyword search. This makes vector databases more flexible than traditional relational databases.</li>
</ul>

<h2 id="conclusion">Conclusion</h2>

<p>Vector databases are a powerful new technology for storing and querying vector representations of data. Vector databases can be used to improve the performance and accuracy of a variety of applications, such as search engines, recommender systems, and machine learning applications.</p>]]></content><author><name>jkfran</name></author><category term="ai" /><summary type="html"><![CDATA[An introduction to vector databases, covering word, image, and audio embeddings, how they work, and when to use them.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/vector-embedding-databases.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/vector-embedding-databases.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Capturing Only Unhandled Exceptions with Sentry in Python</title><link href="https://jkfran.com/capturing-unhandled-exceptions-sentry-python/" rel="alternate" type="text/html" title="Capturing Only Unhandled Exceptions with Sentry in Python" /><published>2023-03-21T00:00:00+00:00</published><updated>2023-03-21T00:00:00+00:00</updated><id>https://jkfran.com/capturing-unhandled-exceptions-sentry-python</id><content type="html" xml:base="https://jkfran.com/capturing-unhandled-exceptions-sentry-python/"><![CDATA[<h2 id="capturing-only-unhandled-exceptions-with-sentry-in-python">Capturing Only Unhandled Exceptions with Sentry in Python</h2>

<p>Sentry is a popular error tracking and monitoring tool that helps developers identify and fix issues in their applications. By default, Sentry captures unhandled exceptions and logged errors. However, in some cases, you might want to focus on unhandled exceptions only, or you might encounter a situation where Sentry reports handled exceptions without your consent from other integrations. In this blog post, we will show you how to configure Sentry to capture unhandled exceptions only in your Python applications.</p>

<blockquote>
  <p>Note: Handled exceptions can be useful in some situations, especially when you want to capture them manually. If you prefer to keep capturing handled exceptions, you can modify the function presented in this post to only ignore logger events.</p>
</blockquote>

<h2 id="configuring-sentry-to-capture-unhandled-exceptions-only">Configuring Sentry to Capture Unhandled Exceptions Only</h2>

<p>To achieve this, you can utilize Sentry’s <code class="language-plaintext highlighter-rouge">before_send</code> callback. This callback allows you to modify and filter events before they are sent to Sentry. Here is the code that demonstrates how to configure Sentry to capture unhandled exceptions only:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
</pre></td><td class="rouge-code"><pre><span class="n">sentry_sdk</span><span class="p">.</span><span class="n">init</span><span class="p">(</span>
    <span class="n">dsn</span><span class="o">=</span><span class="n">dsn</span><span class="p">,</span>
    <span class="n">environment</span><span class="o">=</span><span class="n">environment</span><span class="p">,</span>
    <span class="n">before_send</span><span class="o">=</span><span class="n">sentry_before_send</span><span class="p">,</span>
<span class="p">)</span>

<span class="k">def</span> <span class="nf">sentry_before_send</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">hint</span><span class="p">):</span>
    <span class="s">"""Filters Sentry events before sending.

    This function filters out handled exceptions and logged errors.
    By doing this we will only receive unhandled exceptions on Sentry.

    Args:
        event (dict): The event dictionary containing exception data.

        hint (dict): Additional information about the event, including
            the original exception.

    Returns:
        dict: The modified event dictionary, or None if the event should be
            ignored.
    """</span>

    <span class="c1"># Ignore logged errors
</span>    <span class="k">if</span> <span class="s">"logger"</span> <span class="ow">in</span> <span class="n">event</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">None</span>

    <span class="c1"># Ignore handled exceptions
</span>    <span class="n">exceptions</span> <span class="o">=</span> <span class="n">event</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"exception"</span><span class="p">,</span> <span class="p">{}).</span><span class="n">get</span><span class="p">(</span><span class="s">"values"</span><span class="p">,</span> <span class="p">[])</span>
    <span class="k">if</span> <span class="n">exceptions</span><span class="p">:</span>
        <span class="n">exc</span> <span class="o">=</span> <span class="n">exceptions</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
        <span class="n">mechanism</span> <span class="o">=</span> <span class="n">exc</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"mechanism"</span><span class="p">)</span>

        <span class="k">if</span> <span class="n">mechanism</span><span class="p">:</span>
            <span class="k">if</span> <span class="n">mechanism</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"handled"</span><span class="p">):</span>
                <span class="k">return</span> <span class="bp">None</span>

    <span class="k">return</span> <span class="n">event</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>With this configuration, Sentry will ignore both handled exceptions and logged errors, focusing only on unhandled exceptions. This helps to reduce noise in your Sentry dashboard and allows you to concentrate on the most critical issues in your application.</p>

<p>We hope this post is helpful to those who want to customize Sentry’s exception reporting behavior. Happy debugging!</p>]]></content><author><name>jkfran</name></author><category term="python" /><summary type="html"><![CDATA[How to configure Sentry in Python to capture only unhandled exceptions using the before_send callback.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/sentry-python.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/sentry-python.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Introducing Killport: A Simple CLI Tool to Free Ports in Linux</title><link href="https://jkfran.com/free-ports-linux-killport/" rel="alternate" type="text/html" title="Introducing Killport: A Simple CLI Tool to Free Ports in Linux" /><published>2023-03-19T00:00:00+00:00</published><updated>2023-03-19T00:00:00+00:00</updated><id>https://jkfran.com/free-ports-linux-killport</id><content type="html" xml:base="https://jkfran.com/free-ports-linux-killport/"><![CDATA[<p>Discover how to easily kill processes listening on ports in Linux with Killport</p>

<h2 id="introduction">Introduction</h2>

<p>Today, I am excited to announce the release of a new open-source project called <a href="https://github.com/jkfran/killport">Killport</a>. Killport is a simple command-line interface (CLI) tool designed to help you easily free up ports in Linux. If you’ve ever encountered the issue of a port being occupied by an unknown process or you want to quickly kill a process listening on a specific port, Killport is here to save the day!</p>

<p>In this blog post, we’ll discuss how Killport can help you resolve common port-related issues and demonstrate how to install and use it on your Linux system.</p>

<h2 id="why-killport">Why Killport?</h2>

<p>As developers, we often work with applications that require specific ports to function properly. Occasionally, these ports may be occupied by other processes, causing conflicts and preventing our applications from running smoothly.</p>

<p>Searching for the process that’s listening on a specific port and then killing it can be a cumbersome task, especially when you’re in the middle of development or troubleshooting. That’s where Killport comes in. It simplifies the process of freeing up a port by automatically finding and terminating the process that’s occupying it.</p>

<h2 id="installing-killport">Installing Killport</h2>

<p>The easiest way to install Killport on your Linux system is by running the following command:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>curl <span class="nt">-sL</span> https://bit.ly/killport | sh
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This command will download the Killport installation script and execute it, installing the Killport binary in your <code class="language-plaintext highlighter-rouge">$HOME/.local/bin</code> directory.</p>

<p>You can also find binary releases for various Linux architectures on the <a href="https://github.com/jkfran/killport/releases">Killport GitHub releases page</a>.</p>

<h2 id="using-killport">Using Killport</h2>

<p>Using Killport is straightforward. Simply run the following command, replacing <code class="language-plaintext highlighter-rouge">&lt;port&gt;</code> with the port number you want to free:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>killport &lt;port&gt;
</pre></td></tr></tbody></table></code></pre></div></div>

<p>For example, to kill the process listening on port 8080, you would run:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>killport 8080
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Killport will then identify the process occupying the specified port and terminate it, freeing up the port for your application to use.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Killport is a handy CLI tool that makes it easy to free up ports in Linux by killing the processes listening on them. By simplifying this task, Killport saves you time and helps you maintain a smooth development workflow. Give it a try, and let us know what you think!</p>

<p>Don’t forget to star and contribute to the <a href="https://github.com/jkfran/killport">Killport GitHub repository</a> if you find it useful!</p>]]></content><author><name>jkfran</name></author><category term="linux" /><summary type="html"><![CDATA[Introducing Killport, an open-source Rust CLI tool to easily kill processes listening on specific ports in Linux.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/killport.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/killport.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How to Choose the Right Virtual Machine for Your Kubernetes Cluster on Azure</title><link href="https://jkfran.com/choose-right-virtual-machine-kubernetes-cluster-azure/" rel="alternate" type="text/html" title="How to Choose the Right Virtual Machine for Your Kubernetes Cluster on Azure" /><published>2023-03-14T00:00:00+00:00</published><updated>2023-03-14T00:00:00+00:00</updated><id>https://jkfran.com/choose-right-virtual-machine-kubernetes-cluster-azure</id><content type="html" xml:base="https://jkfran.com/choose-right-virtual-machine-kubernetes-cluster-azure/"><![CDATA[<p>As a professional working in the tech industry, I know firsthand the importance of selecting the right virtual machine (VM) for your workload. Recently, in my current role, I was tasked with selecting the best VM for our Kubernetes cluster on Microsoft Azure. As I dove into my research, I discovered that there were several factors to consider, including memory, CPU, storage, and networking requirements, as well as operating system compatibility and cost.</p>

<p>In this blog post, I’ll share my findings and insights on the different types of Azure virtual machines available and how to select the best one for your workload. Whether you’re new to Azure or a seasoned user, this guide will provide you with valuable information to help you optimize your performance and cost efficiency on the cloud. So, let’s get started!</p>

<h2 id="introduction">Introduction</h2>

<p>Microsoft Azure offers a range of virtual machines (VMs) to suit a variety of computing needs. There are several factors to consider when selecting the best VM for your workload, including memory, CPU, storage, and networking requirements, as well as operating system compatibility and cost.</p>

<p>Here’s a brief overview of the machine types available on Azure:</p>

<ul>
  <li>
    <p>General Purpose: These machines are designed for a wide range of workloads and are available in several series, including B, Dsv3, and Dasv4. They offer a balance of CPU, memory, and network resources at an affordable cost.</p>
  </li>
  <li>
    <p>Compute Optimized: These machines are optimized for high-performance computing workloads and are available in several series, including Fsv2 and Fs. They offer the highest CPU-to-memory ratio and are ideal for compute-intensive applications.</p>
  </li>
  <li>
    <p>Memory Optimized: These machines are designed for memory-intensive workloads and are available in several series, including Esv3, Easv4, and M. They offer a high memory-to-CPU ratio and are ideal for data analytics and in-memory databases.</p>
  </li>
  <li>
    <p>Storage Optimized: These machines are optimized for storage-intensive workloads and are available in several series, including Ls and H. They offer high disk throughput and are ideal for applications that require large-scale storage solutions.</p>
  </li>
  <li>
    <p>GPU: These machines are designed for graphics-intensive workloads and are available in several series, including NV and NC. They offer dedicated GPU resources and are ideal for applications that require high-performance graphics processing.</p>
  </li>
</ul>

<h2 id="choosing-the-best-vm-based-on-your-requirements-for-kubernetes-clusters">Choosing the Best VM based on Your Requirements for Kubernetes Clusters</h2>

<p>When selecting a VM for your Kubernetes cluster on Azure, it’s important to choose one that meets your specific requirements. Here are some factors to consider when choosing the best VM for your workload:</p>

<h3 id="1-memory-and-cpu-requirements">1. Memory and CPU requirements</h3>

<p>Memory and CPU are two of the most important resources to consider when choosing a VM for your Kubernetes cluster. In general, Kubernetes clusters require a high amount of memory and CPU resources to run efficiently. When selecting a VM, make sure it has enough memory and CPU resources to handle your workload.</p>

<h3 id="2-storage-requirements">2. Storage requirements</h3>

<p>Storage requirements are another important consideration when selecting a VM for your Kubernetes cluster. Kubernetes clusters require storage for both the operating system and the container images. When selecting a VM, make sure it has enough storage capacity to handle your workload.</p>

<h3 id="3-networking-requirements">3. Networking requirements</h3>

<p>Networking requirements are also important when selecting a VM for your Kubernetes cluster. Kubernetes clusters require high network bandwidth to handle the communication between nodes and pods. When selecting a VM, make sure it has a high network bandwidth capacity.</p>

<h3 id="4-operating-system-compatibility">4. Operating system compatibility</h3>

<p>Make sure the VM you choose is compatible with your Kubernetes cluster. Azure offers several Kubernetes solutions, including Azure Kubernetes Service (AKS) and Azure Red Hat OpenShift (ARO), each with different operating system requirements.</p>

<h3 id="5-cost">5. Cost</h3>

<p>Cost is also an important consideration when selecting a VM for your Kubernetes cluster. Consider the cost of the VM, as well as any additional costs for storage, networking, and other services.</p>

<h3 id="recommendations">Recommendations</h3>

<p>When selecting a VM for your Kubernetes cluster, it’s important to choose one that meets your specific requirements. If you’re unsure which VM to choose, start with a General Purpose VM, which offers a good balance of CPU, memory, and network resources at an affordable cost. If your Kubernetes cluster requires high-performance computing, choose a Compute Optimized VM. If your Kubernetes cluster requires a high amount of memory, choose a Memory Optimized VM. If your Kubernetes cluster requires a large amount of storage, choose a Storage Optimized VM. Finally, if your Kubernetes cluster requires high-performance graphics processing, choose a GPU Optimized VM.</p>

<p>Keep in mind that you can always scale up or down your VM based on your Kubernetes cluster requirements. Azure also offers several tools and services, such as AKS Advisor, that can help you optimize your VM configuration for better performance and cost savings.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Choosing the right virtual machine (VM) for your workload is crucial for optimizing performance and cost efficiency. Microsoft Azure offers a range of VM types to meet different computing needs, including General Purpose, Compute Optimized, Memory Optimized, Storage Optimized, and GPU Optimized machines.</p>

<p>When selecting a VM, it’s important to consider your specific requirements, such as memory, CPU, storage, and networking needs, as well as operating system compatibility and cost. For Kubernetes clusters, in addition to the above factors, high network bandwidth is also a critical requirement.</p>

<p>Based on your specific workload needs, you can select the appropriate VM type, scale up or down as needed, and optimize your configuration for better performance and cost savings. Azure offers several tools and services to help you do this, such as AKS Advisor for Kubernetes clusters.</p>

<p>In summary, choosing the right VM for your workload is essential for achieving optimal performance and cost efficiency on Microsoft Azure. With careful consideration of your requirements and the available VM types, you can select the best VM for your workload and achieve the best possible outcomes.</p>]]></content><author><name>jkfran</name></author><category term="devops" /><summary type="html"><![CDATA[A comprehensive guide to selecting Azure VM types for Kubernetes clusters, covering memory, CPU, storage, networking, and cost.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/azure-kubernetes-vm.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/azure-kubernetes-vm.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How to run Redis locally with Docker</title><link href="https://jkfran.com/run-redis-locally-docker/" rel="alternate" type="text/html" title="How to run Redis locally with Docker" /><published>2022-12-01T00:00:00+00:00</published><updated>2022-12-01T00:00:00+00:00</updated><id>https://jkfran.com/run-redis-locally-docker</id><content type="html" xml:base="https://jkfran.com/run-redis-locally-docker/"><![CDATA[<p>In this post, I want to share just one command to run Redis locally. That’s it, and it’ll be quick and easy.</p>

<h2 id="requirements">Requirements</h2>

<ul>
  <li>Docker (<a href="https://docs.docker.com/get-docker/">https://docs.docker.com/get-docker/</a>)</li>
</ul>

<h2 id="get-and-run-redis">Get and run Redis</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>docker run <span class="nt">-it</span> <span class="nt">-p</span> 6379:6379 <span class="nt">--rm</span> <span class="nt">--name</span> my-redis redis
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That’s it!</p>

<p><img src="https://user-images.githubusercontent.com/6353928/205104552-7ae20743-10ae-4f7f-a521-d81d16291074.png" alt="Redis CLI connected to Docker container" width="1472" height="556" loading="lazy" /></p>

<p>Note: If you want to run something more customized, I recommend looking into the <a href="https://redis.io/docs/">Redis documentation</a>.</p>

<h2 id="redis-client">Redis client</h2>

<p>I didn’t have the opportunity to work with Redis before, so when I looked at the CLI client, the commands were completely new to me.
If you are looking for an easy way to view the data. I recommend <a href="https://github.com/tiagocoutinho/qredis">QRedis</a>.</p>

<h3 id="installation">Installation</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="nb">sudo </span>pip3 <span class="nb">install </span>qredis
</pre></td></tr></tbody></table></code></pre></div></div>

<h4 id="connecting">Connecting</h4>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>qredis <span class="nt">-p</span> 6379
</pre></td></tr></tbody></table></code></pre></div></div>

<p><img src="https://user-images.githubusercontent.com/6353928/205104704-e6d2f91a-df9d-41c7-a69c-102410f61417.png" alt="RedisInsight GUI showing keys" width="769" height="598" loading="lazy" /></p>

<p>Ah! Much better.</p>]]></content><author><name>jkfran</name></author><category term="devops" /><summary type="html"><![CDATA[A quick guide to running Redis in Docker with a single command and using the QRedis GUI client for easier data management.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/redis-docker.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/redis-docker.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">I built a little driving arcade machine with a Raspberry Pi</title><link href="https://jkfran.com/built-driving-arcade-machine-raspberry-pi/" rel="alternate" type="text/html" title="I built a little driving arcade machine with a Raspberry Pi" /><published>2022-10-07T00:00:00+00:00</published><updated>2022-10-07T00:00:00+00:00</updated><id>https://jkfran.com/built-driving-arcade-machine-raspberry-pi</id><content type="html" xml:base="https://jkfran.com/built-driving-arcade-machine-raspberry-pi/"><![CDATA[<p>Hello! This summer, I went to Spain as usual and decided to stay longer for different reasons. I wanted to occupy my mind and build something fun during this time. I also wanted to involve my nieces and do something for them, so they remember me when I am away.</p>

<p>I started looking around in my parents’ house,
it’s a big house with a lot of space, enough to keep things that nobody needs anymore,
so I went to the garage, and I found this old screen:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194565997-42e04ec8-6fe3-4800-a241-b45579cfc2be.png" alt="Arcade machine wooden frame" width="765" height="574" loading="lazy" /></p>

<p>I used to use this old screen, but the support was broken, so there was no way to keep it straight.</p>

<p>I also thought I had a Raspberry Pi so that I could build some kind of video game.
Here is my old Raspberry Pi 1:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194566138-7e9ebafd-1b6e-4a8e-a4b4-514bfa31bd07.png" alt="Frame construction detail" width="765" height="737" loading="lazy" /></p>

<p>I decided to create a little game on the raspberry pi involving my niece in the process,
I felt like I wanted to teach her that anybody can be an engineer and create stuff.</p>

<p>We dedicated some time to building our own game. I wanted to start with something simple, so the following code for the pong game was a good start:
<a href="https://github.com/Wireframe-Magazine/Code-the-Classics/blob/master/boing-master/boing.py">github.com/Wireframe-Magazine/Code-the-Classics/blob/master/boing-master/boing.py</a></p>

<p><img src="https://user-images.githubusercontent.com/6353928/194565167-25fb1857-dac9-4e8f-bacd-5613034129bc.png" alt="Monitor mounted in frame" width="765" height="574" loading="lazy" /></p>

<p>We ended up making many changes to the game, and my niece even wanted to change the bars for 2D warriors:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194565506-f02dff38-5fb2-4141-84ed-414a85405ecd.png" alt="Steering wheel assembly" width="798" height="599" loading="lazy" /></p>

<p>It turns out the Raspberry Pi 1 was super slow, even with overclocking.
I was not surprised. When I bought it, it was a revolution but not anymore.
The Raspberry was struggling a bit to run this simple game, but we had a lot of fun just doing the game, so I didn’t care much.</p>

<p>When she left, I knew I had two options:</p>

<ul>
  <li>Stop the adventure here</li>
  <li>Build this pong machine and find two remotes for two players</li>
</ul>

<p>I am not going to lie; I was as excited as her to finish it.
I started looking around at home, and I found just one old PS1 controller and a PS-USB adapter, but the controller was broken :(</p>

<p>My next finding was the item that made me change course and build a driving arcade instead of the pong machine:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194569343-c3ad6fa3-c05d-4efe-ab26-fc2936fa0d0f.png" alt="Raspberry Pi wiring" width="765" height="383" loading="lazy" /></p>

<p>It’s an old driving controller for PlayStation 1.
I also found this old stool that I decided was going to be the machine body:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194569868-dbaf582d-c43b-4460-a2bf-310b5e979021.png" alt="Pedals mechanism" width="765" height="759" loading="lazy" /></p>

<p>I knew that building this kind of machine would require more computational power so I had to buy the latest Raspberry Pi 4 to finish this project,
setting up the controls was a nightmare, but once I was done, I got a few games on Retropie
and set up the Kids mode on it:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194570670-fd76c551-4f00-4d37-ac47-eb186d0b2325.png" alt="Arcade machine painted" width="765" height="574" loading="lazy" /></p>

<p>After some wooden and painting work, the machine was ready to use. Here are some pictures with the kids playing on it:</p>

<p><img src="https://user-images.githubusercontent.com/6353928/194571251-88132b4a-ae5d-4f42-afec-775859bd197b.png" alt="Completed arcade cabinet front" width="765" height="951" loading="lazy" /></p>

<p><img src="https://user-images.githubusercontent.com/6353928/194571553-3e70d6bb-61f1-4cee-b522-4e48e0d0b68d.png" alt="Arcade machine side view" width="765" height="582" loading="lazy" /></p>

<p><img src="https://user-images.githubusercontent.com/6353928/194571884-893fbe1b-000a-49cd-86e5-d9759dd60c38.png" alt="Finished arcade machine with game running" width="765" height="574" loading="lazy" /></p>

<p>I regret not taking more photos of the machine to show how everything fit, cables, connections, etc. The final result was tidy and the only cable getting out of the device was the power cable.</p>

<p>The only item I had to buy was the Raspberry PI. Everything else was in my parent’s house, which is crazy. It took me about three or four days to finish everything.</p>

<p>Thank you for reading!</p>]]></content><author><name>jkfran</name></author><category term="projects" /><summary type="html"><![CDATA[The story of building a driving arcade game on a Raspberry Pi 1 with my niece, from modifying a Pong game to sourcing hardware.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/raspberry-pi-arcade.jpg" /><media:content medium="image" url="https://github.com/jkfran/jkfran.com/releases/download/blog-images/raspberry-pi-arcade.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>