8. Dynamic

The interfaces we design, for now, are static. We cannot copy, delete or move widgets on the fly. Which is a problem if we want to design living, dynamic interfaces. For example, I have a product search. Statically I create a text field and a button, which are going to be there from the start. Now, when the user clicks the button, I want to capture the text to run a search and show a list of widgets with the results. How do I do it? How do I delete the text field and the button? How do I create the complex widgets with the results without restarting the application? How do I delete the results to run a new search? These and other questions we will answer in this lesson.

To illustrate how it works, we will make requests to a fictional REST API (dummyjson) to search for products.

In the following video you can see the final result.

Therefore, in this lesson we will learn to:

• Create widgets dynamically.
• Delete existing widgets.
• Navigate between layouts.
• Make HTTP requests to a REST API.
• Create keyboard shortcuts to navigate through the results.

Let's go for it!

1. Creating the main layout

To create the UI we will use the widget package, as on other occasions.

;; -*- coding: utf-8 -*-
(require 'widget)

We will define the minimal variables.

(defvar buffer-name "*Search products*")
(defvar separator "------------------")
(defvar input-name)
(defvar found-products)
(defvar list-products '())

• buffer-name: Name of the buffer where the UI will be shown. We will use it to know which buffer we have to kill when we want to close the UI.
• separator: Separator we will use to separate the products.
• input-name: Variable where we will store the name input widget in which the user will enter the name of the product to search for.
• found-products: Variable where we will store the widget that will show the number of products found.
• list-products: List where we will store the widgets of the found products that we will show in the UI.

We will create the main-layout function that will be in charge of rendering the UI.

(defun main-layout ()
  "Make widgets for the main layout."
  (interactive)
  ;; Clear variables
  (setq input-name nil)
  (setq found-products nil)
  (setq list-products '())
  ;; Create the buffer
  (switch-to-buffer buffer-name)
  ;; Clear the buffer
  (kill-all-local-variables)
  (let ((inhibit-read-only t))
    (erase-buffer))
  (remove-overlays)
  ;; Create the widgets
  ;; Title
  (widget-insert "- Search products -\n\n")
  ;; Input name
  (setq input-name (widget-create 'editable-field
                  :size 15
                  :tag "name"
                  :help-echo "Type a name"
                  :format "Name: %v"))
  ;; Separator
  (widget-insert " ")
    ;; Button search
  (widget-create 'push-button
         :notify #'search-products
         :help-echo "Search products"
         :highlight t
         :button-face '(:background "gray" :foreground "black")
         "Search")
  (widget-insert "\n")
  ;; End widgets
  ;; Display the buffer
  (use-local-map widget-keymap)
  (widget-setup)
  ;; Go to the first input field
  (widget-forward 1))

The most notable part is where we rewrite certain variables to nil. We do this so that when we call the main-layout function a second time the widgets are not duplicated, or so that while we are developing we do not have to clear the buffer to run the function again.

All together it would look like this.

;; -*- coding: utf-8 -*-
;; Imports
(require 'widget)

(eval-when-compile
  (require 'wid-edit))

;; Variables
(defvar buffer-name "*Search products*")
(defvar separator "------------------")
(defvar input-name)
(defvar found-products)
(defvar list-products '())

;; Functions

(defun main-layout ()
  "Make widgets for the main layout."
  (interactive)
  ;; Clear variables
  (setq input-name nil)
  (setq found-products nil)
  (setq list-products '())
  ;; Create the buffer
  (switch-to-buffer buffer-name)
  ;; Clear the buffer
  (kill-all-local-variables)
  (let ((inhibit-read-only t))
    (erase-buffer))
  (remove-overlays)
  ;; Create the widgets
  ;; Title
  (widget-insert "- Search products -\n\n")
  ;; Input name
  (setq input-name (widget-create 'editable-field
                  :size 15
                  :tag "name"
                  :help-echo "Type a name"
                  :format "Name: %v"))
  ;; Separator
  (widget-insert " ")
    ;; Button search
  (widget-create 'push-button
         :notify #'search-products
         :help-echo "Search products"
         :highlight t
         :button-face '(:background "gray" :foreground "black")
         "Search")
  (widget-insert "\n")
  ;; End widgets
  ;; Display the buffer
  (use-local-map widget-keymap)
  (widget-setup)
  ;; Go to the first input field
  (widget-forward 1))


;; Initialization
(main-layout)

2. Searching for products

To search for products we will use the REST API of dummyJSON.com. This API lets us get fake data in JSON format. In our case, we will use the /products/search route, which gives us the possibility to filter products by name.

The request consists of the endpoint /products/search and two parameters:

• q: Name of the product to search for.
• limit: Number of products to show. If we put 0 it will return all the products.

The equivalent in curl would be:

curl -X GET "https://dummyjson.com/products/search?q=phone&limit=0" -H "accept: application/json"

To make the request in Elisp we will use the request package. This package lets us make HTTP requests easily. If you do not have it installed, you can install it with M-x package-install RET request RET.

(request "https://dummyjson.com/products/search"
    :params `(("q" . ,(widget-value input-name))
          ("limit" . "0"))
    :parser 'json-read
    :sync t
    :success (cl-function
              (lambda (&key data &allow-other-keys)
              ;; Data
                )))

3. Inserting and deleting widgets dynamically

We cannot create the product widgets without first understanding that they are volatile elements, which later we will need to delete when we run a new search. Any element we insert dynamically, we will have to store its reference so we can remove it later.

I will create a function to create the widget of a product.

(defun insert-product (item)
  "Render product."
  (add-to-list 'list-products (widget-create 'item
                         :format (format-product item)
                         :value (assoc-default 'id item))))

Each new product, or item, will be stored in list-products so as not to lose the memory reference. The (format-product) function is in charge of formatting the product text, or the format with the fields.

(defun format-product (item)
  "Format product."
  (format "\n%s\n%s\n%s\nPrice: %s€ Discount: %s%%"
      separator
      (assoc-default 'title item)
      (assoc-default 'description item)
      (assoc-default 'price item)
      (assoc-default 'discountPercentage item)))

We will also need another function to insert the number of products found.

(defun insert-number-of-products ()
  "Insert number of products."
  (setq found-products (widget-create 'item
                      :format "\nFound %v products\n"
                      :value (length request-products))))

Likewise, we will store its reference in a variable.

Finally we will make use of a function to clear, or delete, the widgets. Before inserting the new results, we must get rid of the previous ones.

(defun clear-results ()
  "Clear list-products."
  ;; Clear found-products
  (when (not (eq found-products nil)) (widget-delete found-products))
  ;; Clear list-products
  (dolist (list-item list-products)
    (widget-delete list-item))
  (setq list-products '()))

If we join all the pieces we will already have a function in charge of getting the products from the API, deleting the previous widgets and rendering new ones.

(defun search-products (widget &rest ignore)
  "Search products in dummyJSON.com."
  ;; Cursor to end of buffer
  (goto-char (point-max))
  ;; Show loading message
  (message "Searching products...")
  ;; Request data from dummyJSON.com
  (request "https://dummyjson.com/products/search"
    :params `(("q" . ,(widget-value input-name))
          ("limit" . "0"))
    :parser 'json-read
    :sync t
    :success (cl-function
              (lambda (&key data &allow-other-keys)
        (let ((request-products (assoc-default 'products data)))
          (clear-results)
          (insert-number-of-products)
          ;; Add products to list-products
          (cl-loop for item across request-products
               do (insert-product item)))
        ;; Focus to button search
        (widget-forward -1)))))

To the question of why the synchronous request has been enabled instead of the asynchronous one, :sync t, it is because we do not want the user to be able to run a new search until the previous one has completed. Otherwise, we could have concurrency problems. It is easy to fix using a control or state variable.

4. Navigating through the results with keyboard shortcuts

To navigate through the results, or products, we will use keyboard shortcuts. Specifically the n key to go to the next product and the p key to go to the previous product.

(define-key widget-keymap (kbd "n") (lambda ()
                      (interactive)
                      (search-forward separator)
                      (forward-line 1)))
(define-key widget-keymap (kbd "p") (lambda ()
                        (interactive)
                        (search-backward separator)
                        (search-backward separator)
                        (forward-line 1)))

While we are at it, we will also add a keyboard shortcut to close the UI with the q key.

(define-key widget-keymap (kbd "q") (lambda ()
                      (interactive)
                      (kill-buffer buffer-name)))

5. Including images

We will need a function capable of downloading the image, through a valid URL, and positioning it in the buffer with the position and size we want. We can rely on the lesson where we talked about images.

(defun put-image-from-url (url &optional width pos)
  "Put an image from an URL in the buffer at position."
  (unless pos (setq pos (1+ (count-lines 1 (point)))))
  (unless url (setq url (url-get-url-at-point)))
  (unless url
    (error "Couldn't find URL."))
  (let ((buffer (url-retrieve-synchronously url)))
    (unwind-protect
        (let ((data (with-current-buffer buffer
                      (goto-char (point-min))
                      (search-forward "\n\n")
                      (buffer-substring (point) (point-max)))))
      (save-excursion
            (goto-char (point-min))
            (forward-line (1- pos)) ; Go to the beginning of the specified line
            (setq pos (line-beginning-position)))
      (put-image (create-image data nil t :width width) pos))
      (kill-buffer buffer))))

Next we call the function when rendering the product in the insert-product function.

(defun insert-product (item)
  "Render product."
  ;; Add text
  (add-to-list 'list-products (widget-create 'item
                         :format (format-product item)
                         :value (assoc-default 'id item)))
  ;; Add image
  (goto-char (point-max))
  (when (search-backward separator nil t)
    (beginning-of-line))
  (forward-line 2)
  (put-image-from-url (assoc-default 'thumbnail item) 200) ;; New
  (goto-char (point-max)))

We will not have a widget to help us, but we can create the mechanism ourselves.

6. Informing the user with a loading layout

When we run a search, the user does not know whether the application is working or not. Therefore, it is a good idea to show a message with a minimum of feedback. For that, we will create a new layout that will be displayed while the request is being made. We are going to learn to navigate between layouts. The strategy is simple: switch to a new buffer with the new layout. When we receive the results from the API, we will delete the buffer, go back to the main layout, clear old results and render the new ones.

First we include some new variables.

(defvar loading--name-buffer "*Loading*")
(defvar loading-text "Loading")

• loading--name-buffer: Name of the buffer where the loading message will be shown.
• loading-text: Text of the loading message.

We do not have any means to center a text horizontally and vertically in a buffer. However, we can calculate the horizontal and vertical padding needed for it. In other words, we will calculate the number of blank spaces or line breaks we need to center the text.

(defun loading--horizontal-padding ()
  "Calculate the horizontal padding for the loading text."
  (let* ((buffer-width (window-width))
     (text-length (length loading-text))
         (horizontal-padding (/ (- buffer-width text-length) 2)))
    (make-string (max 0 horizontal-padding) ?\s)))

(defun loading--vertical-padding ()
  "Calculate the vertical padding for the loading text."
  (let* ((buffer-height (window-height))
     (vertical-padding (/ (- buffer-height 1) 2))) ;; Subtract 1 for the mode line
    (make-string (max 0 vertical-padding) ?\n)))

In addition we will include a function to format the text with the calculated padding.

(defun loading--format-text ()
  "Format the loading text with padding."
  (format "%s%s%s" (loading--vertical-padding) (loading--horizontal-padding) loading-text))

The next thing will be to create the new layout.

(defun loading-layout ()
  "Create the main layout for the loading screen."
  (switch-to-buffer loading--name-buffer)
  (read-only-mode 1)
  (kill-all-local-variables)
  (let ((inhibit-read-only t))
    (erase-buffer))
  (remove-overlays)
  (erase-buffer)
  (widget-create 'item :value (loading--format-text))
  (use-local-map widget-keymap)
  (widget-setup)
  (display-line-numbers-mode 0))

The only notable line is (widget-create 'item :value (loading--format-text)). It creates a widget of type item with the formatted text.

In addition we will need functions to show or hide the loading message.

(defun loading--show ()
  "Show the loading screen."
  (loading-layout))

(defun loading--hide ()
    "Hide the loading screen."
    (kill-buffer loading--name-buffer))

We now have all the tools. All that is left is to decide when we will show the loading message and when we will hide it. In this case, we will show it before making the request and we will hide it when we receive the results.

(defun search-products (widget &rest ignore)
  "Search products in dummyJSON.com."
  (goto-char (point-max))
  (loading--show) ;; We show it
  (request "https://dummyjson.com/products/search"
    :params `(("q" . ,(widget-value input-name))
          ("limit" . "0"))
    :parser 'json-read
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
        (let ((request-products (assoc-default 'products data)))
          (loading--hide) ;; We hide it
          (clear-results)
          (insert-number-of-products)
          (cl-loop for item across request-products
               do (insert-product item)))
        (widget-forward -1)))))

Our loading is now ready.

7. Running the application

To run the application we only have to call the main-layout function. Additionally, we can include the get-separator function to calculate the width of the separator dynamically.

(defun get-separator (&optional separator)
  (let* ((sep (or separator ?─))
     ;;(size (window-max-chars-per-line))
     (size 30)
     (line (make-string size sep)))
    line))

;; Initialization
(setq separator (get-separator))
(main-layout)

Complete example

The source code of the example is below:

;; -*- coding: utf-8 -*-
;; Imports
(require 'widget)
(require 'cl-lib)
(require 'url)

(eval-when-compile
  (require 'wid-edit))

;; Variables
(defvar buffer-name "*Search products*")
(defvar loading--name-buffer "*Loading*")
(defvar loading-text "Loading")
(defvar separator "")
(defvar input-name)
(defvar input-is-discount)
(defvar found-products)
(defvar list-products '())

;; Functions

(defun get-separator (&optional separator)
  (let* ((sep (or separator ?─))
     ;;(size (window-max-chars-per-line))
     (size 30)
     (line (make-string size sep)))
    line))

(defun loading--horizontal-padding ()
  "Calculate the horizontal padding for the loading text."
  (let* ((buffer-width (window-width))
     (text-length (length loading-text))
         (horizontal-padding (/ (- buffer-width text-length) 2)))
    (make-string (max 0 horizontal-padding) ?\s)))

(defun loading--vertical-padding ()
  "Calculate the vertical padding for the loading text."
  (let* ((buffer-height (window-height))
     (vertical-padding (/ (- buffer-height 1) 2))) ;; Subtract 1 for the mode line
    (make-string (max 0 vertical-padding) ?\n)))

(defun loading--format-text ()
  "Format the loading text with padding."
  (format "%s%s%s" (loading--vertical-padding) (loading--horizontal-padding) loading-text))

(defun loading--show ()
  "Show the loading screen."
  (loading-layout))

(defun loading--hide ()
  "Hide the loading screen."
  (kill-buffer loading--name-buffer))


(defun put-image-from-url (url &optional width pos)
  "Put an image from an URL in the buffer at position."
  (unless pos (setq pos (1+ (count-lines 1 (point)))))
  (unless url (setq url (url-get-url-at-point)))
  (unless url
    (error "Couldn't find URL."))
  (let ((buffer (url-retrieve-synchronously url)))
    (unwind-protect
        (let ((data (with-current-buffer buffer
                      (goto-char (point-min))
                      (search-forward "\n\n")
                      (buffer-substring (point) (point-max)))))
      (save-excursion
            (goto-char (point-min))
            (forward-line (1- pos)) ; Go to the beginning of the specified line
            (setq pos (line-beginning-position)))
      (put-image (create-image data nil t :width width) pos))
      (kill-buffer buffer))))

(defun insert-number-of-products ()
  "Insert number of products."
  (setq found-products (widget-create 'item
                      :format "\nFound %v products\n"
                      :value (length request-products))))

(defun clear-results ()
  "Clear list-products."
  ;; Clear all images
  (remove-images (point-min) (point-max))
  ;; Clear found-products
  (when (not (eq found-products nil)) (widget-delete found-products))
  ;; Clear list-products
  (dolist (list-item list-products)
    (widget-delete list-item))
  (setq list-products '()))

(defun format-product (item)
  "Format product."
  (format "\n%s\n\n\n\n🔸 %s 🔸\n📖 %s\n💰: %s€\n🏷️: %s%%"
      separator
      (assoc-default 'title item)
      (assoc-default 'description item)
      (assoc-default 'price item)
      (assoc-default 'discountPercentage item)))

(defun insert-product (item)
  "Render product."
  ;; Add text
  (add-to-list 'list-products (widget-create 'item
                         :format (format-product item)
                         :value (assoc-default 'id item)))
  ;; Add image
  (goto-char (point-max))  ; Start from the beginning of the buffer
  (when (search-backward separator nil t) ; Search for the "important" string
    (beginning-of-line))
  (forward-line 2)
  (put-image-from-url (assoc-default 'thumbnail item) 200)
  (goto-char (point-max)))

;;https://dummyjson.com/products/search?q=text&limit=0
(defun search-products (widget &rest ignore)
  "Search products in dummyJSON.com."
  ;; Cursor to end of buffer
  (goto-char (point-max))
  ;; Show loading
  (loading--show)
  ;; Request data from dummyJSON.com
  (request "https://dummyjson.com/products/search"
    :params `(("q" . ,(widget-value input-name))
          ("limit" . "0"))
    :parser 'json-read
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
        (let ((request-products (assoc-default 'products data)))
      (loading--hide)
          (clear-results)
          (insert-number-of-products)
          ;; Add products to list-products
          (cl-loop for item across request-products
               do (insert-product item)))
        ;; Focus to button search
        (widget-forward -1)))))

(defun main-layout ()
  "Make widgets for the main layout."
  (interactive)
  ;; Clear variables
  (setq input-name nil)
  (setq input-is-discount nil)
  (setq found-products nil)
  (setq list-products '())
  ;; Create the buffer
  (switch-to-buffer buffer-name)
  ;; Clear the buffer
  (kill-all-local-variables)
  (let ((inhibit-read-only t))
    (erase-buffer))
  (remove-overlays)
  ;; Create the widgets
  ;; Title
  (widget-insert "\n Search products 🔎\n\n")
  ;; Input name
  (setq input-name (widget-create 'editable-field
                  :size 15
                  :tag "name"
                  :help-echo "Type a name"
                  :format "Name: %v"))
  ;; Separator
  (widget-insert " ")
    ;; Button search
  (widget-create 'push-button
         :notify #'search-products
         :help-echo "Search products"
         :highlight t
         :button-face '(:background "gray" :foreground "black")
         "Search")
  (widget-insert "\n")
  ;; End widgets
  ;; Display the buffer
  (use-local-map widget-keymap)
  (widget-setup)
  (display-line-numbers-mode 0)
  ;; Go to the first input field
  (widget-forward 1))

(defun loading-layout ()
  "Create the main layout for the loading screen."
  (switch-to-buffer loading--name-buffer)
  (read-only-mode 1)
  (kill-all-local-variables)
  (let ((inhibit-read-only t))
    (erase-buffer))
  (remove-overlays)
  (erase-buffer)
  (widget-create 'item :value (loading--format-text))
  (use-local-map widget-keymap)
  (widget-setup)
  (display-line-numbers-mode 0))


;; Controls
;; n - Next item
;; p - Previous item
;; q - Quit
(define-key widget-keymap (kbd "n") (lambda ()
                      (interactive)
                      (search-forward separator)
                      (forward-line 1)))
(define-key widget-keymap (kbd "p") (lambda ()
                      (interactive)
                      (search-backward separator)
                      (search-backward separator)
                      (forward-line 1)))
(define-key widget-keymap (kbd "q") (lambda ()
                      (interactive)
                      (kill-buffer buffer-name)))

;; Initialization
(setq separator (get-separator))
(main-layout)

Possible improvements you can implement

It is a very simple example; there is room for many controls and helpers in the interface. Some of them are:

• Paginate the results, where we limit the number of products and have buttons to travel between the pages.
• Replace the type of the products with link-url to be able to navigate to the product page.
• More advanced search filters (price, discount, ratings, etc.).

All of these will be small challenges that will help you improve your skills in Emacs Lisp.

Activity 1

Create a calculator to obtain the body mass index (BMI). The fields will be split into steps, each step being a buffer. Or put another way, instead of showing a form with all the fields, we will be asked for each piece of data in a separate buffer. In the last buffer the result will be displayed.

To calculate the BMI we need the following data:

• Height (cm)
• Weight (kg)

Look up the formula to calculate the BMI on the web.

Include buttons to move forward or backward between the steps.

Activity 2

Program a Mastodon profile viewer.

The user will only enter the account name.

Research the Mastodon API endpoint to get the data of a user and their posts.

Include images, links and buttons to navigate through their post history.