Data Guide #

The data structures are one of the most complex and important parts of Penpot. It's critical that the data integrity is always maintained throughout the whole usage, and also file exports & imports and data model evolution.

To modify the data structure (the most typical case will be to add a new attribute to the shapes), this list must be checked. This is not an exhaustive list, but all of this is important in general.

General considerations #

• We prefer that the page and shape attributes are optional. I.E. there is a default object behavior, that occurs when the attribute is not present, and its presence activates some feature (example: if there is no fill-color, the shape is not filled). When you revert to the default state, it's better to remove the attribute than leaving it with null value. There are some process (for example import & export) that filter out and remove all attributes that are null.

• So never expect that attribute with null value is a different state that without the attribute.

• In objects attribute names we don't use special symbols that are allowed by Clojure (for example ending it with ? for boolean values), because this may cause problems when exporting.

Code organization in abstraction levels #

Initially, Penpot data model implementation was organized in a different way. We are currently in a process of reorganization. The objective is to have data manipulation code structured in abstraction layers, with well-defined boundaries.

At this moment the namespace structure is already organized as described here, but there is much code that does not comply with these rules, and needs to be moved or refactored. We expect to be refactoring existing modules incrementally, each time we do an important functionality change.

Abstract data types #

▾ common/
  ▾ src/app/common/
    ▾ types/
        file.cljc
        page.cljc
        shape.cljc
        color.cljc
        component.cljc
        ...

Namespaces here represent a single data structure, or a fragment of one, as an abstract data type. Each structure has:

• A schema spec that defines the structure of the type and its values:

  (sm/define! ::fill
    [:map {:title "Fill"}
     [:fill-color {:optional true} ::ctc/rgb-color]
     [:fill-opacity {:optional true} ::sm/safe-number]
     ...)
  
  (sm/define! ::shape-attrs
    [:map {:title "ShapeAttrs"}
     [:name {:optional true} :string]
     [:selrect {:optional true} ::grc/rect]
     [:points {:optional true} ::points]
     [:blocked {:optional true} :boolean]
     [:fills {:optional true}
      [:vector {:gen/max 2} ::fill]]
     ...)

• Helper functions to create, query and manipulate the structure. Helpers at this level only are allowed to see the internal attributes of a type. Updaters receive an object of the type, and return a new object modified, also ensuring the internal integrity of the data after the change.

  (defn setup-shape
   "A function that initializes the geometric data of the shape. The props must
   contain at least :x :y :width :height."
   [{:keys [type] :as props}]
   ...)
  
  (defn has-direction?
  [interaction]
    (#{:slide :push} (-> interaction :animation :animation-type)))
  
  (defn set-direction
  [interaction direction]
    (dm/assert!
     "expected valid interaction map"
     (check-interaction! interaction))
    (dm/assert!
     "expected valid direction"
     (contains? direction-types direction))
    (dm/assert!
     "expected compatible interaction map"
     (has-direction? interaction))
    (update interaction :animation assoc :direction direction))

IMPORTANT: we should always use helper functions to access and modify these data structures. Avoid direct attribute read or using functions like assoc or update, even if the information is contained in a single attribute. This way it will be much simpler to add validation checks or to modify the internal representation of a type, and will be easier to search for places in the code where this data item is used.

Currently much of Penpot code does not follow this requirement, but it should do in new code or in any refactor.

File operations #

▾ common/
  ▾ src/app/common/
    ▾ files/
        helpers.cljc
        shapes_helpers.cljc
        ...

Functions that modify a file object (or a part of it) in place, returning the file object changed. They ensure the referential integrity within the file, or between a file and its libraries.

(defn resolve-component
  "Retrieve the referenced component, from the local file or from a library"
  [shape file libraries & {:keys [include-deleted?] :or {include-deleted? False}}]
  (if (= (:component-file shape) (:id file))
        (ctkl/get-component (:data file) (:component-id shape) include-deleted?)
        (get-component libraries
                      (:component-file shape)
                      (:component-id shape)
                      :include-deleted? include-deleted?)))

(defn delete-component
"Mark a component as deleted and store the main instance shapes inside it, to
be able to be recovered later."
[file-data component-id skip-undelete? Main-instance]
(let [components-v2 (dm/get-in file-data [:options :components-v2])]
  (if (or (not components-v2) skip-undelete?)
        (ctkl/delete-component file-data component-id)
        (let [set-main-instance ;; If there is a saved main-instance, restore it.
               #(if main-instance
                  (assoc-in % [:objects (:main-instance-id %)] main-instance)
                  %)]
        (-> file-data
           (ctkl/update-component component-id load-component-objects)
           (ctkl/update-component component-id set-main-instance)
           (ctkl/mark-component-deleted component-id))))))

This module is still needing an important refactor. Mainly to take functions from common.types and move them here.

File validation and repair #

There is a function in app.common.files.validate that checks a file for referential and semantic integrity. It's called automatically when file changes are sent to backend, but may be invoked manually whenever it's needed.

File changes objects #

▾ common/
  ▾ src/app/common/
    ▾ files/
        changes_builder.cljc
        changes.cljc
        ...

Wrap the update functions in file operations module into changes objects, that may be serialized, stored, sent to backend and executed to actually modify a file object. They should not contain business logic or algorithms. Only adapt the interface to the file operations or types.

(sm/define! ::changes
  [:map {:title "changes"}
   [:redo-changes vector?]
   [:undo-changes seq?]
   [:origin {:optional true} any?]
   [:save-undo? {:optional true} boolean?]
   [:stack-undo? {:optional true} boolean?]
   [:undo-group {:optional true} any?]])

(defmethod process-change :add-component
  [file-data params]
  (ctkl/add-component file-data params))

Business logic #

▾ common/
  ▾ src/app/common/
    ▾ logic/
        shapes.cljc
        libraries.cljc

Functions that implement semantic user actions, in an abstract way (independent of UI). They don't directly modify files, but generate changes objects, that may be executed in frontend or sent to backend.

(defn generate-instantiate-component
"Generate changes to create a new instance from a component."
[changes objects file-id component-id position page libraries old-id parent-id
  frame-id {:keys [force-frame?] :or {force-frame? False}}]
 (let [component        (ctf/get-component libraries file-id component-id)
        parent          (when parent-id (get objects parent-id))
        library         (get libraries file-id)
        components-v2 (dm/get-in library [:data :options :components-v2])
        [new-shape new-shapes]º
        (ctn/make-component-instance page
                                        Component
                                        (:data library)
                                        Position
                                        Components-v2
                                        (cond-> {}
                                   force-frame? (assoc :force-frame-id frame-id)))
        changes (cond-> (pcb/add-object changes first-shape {:ignore-touched true})
                 (some? old-id) (pcb/amend-last-change #(assoc % :old-id old-id)))
        changes (reduce #(pcb/add-object %1 %2 {:ignore-touched true})
                       changes
                       (rest new-shapes))]
[new-shape changes]))

Data migrations #

▾ common/
  ▾ src/app/common/
    ▾ files/
        migrations.cljc

When changing the model it's essential to take into account that the existing Penpot files must keep working without changes. If you follow the general considerations stated above, usually this is automatic, since the objects already in the database just have the default behavior, that should be the same as before the change. And the new features apply to new or edited objects.

But if this is not possible, and we are talking of a breaking change, you can write a data migration. Just define a new data version and a migration script in migrations.cljc and increment file-version in common.cljc.

From then on, every time a file is loaded from the database, if its version number is lower than the current version in the app, the file data will be handled to all the needed migration functions. If you later modify and save the file, it will be now updated in database.

Shape edit forms #

▾ frontend/
  ▾ src/
    ▾ app/
      ▾ main/
        ▾ ui/
          ▾ workspace/
            ▾ sidebar/
              ▾ options/
                ▸ menus/
                ▸ rows/
                ▾ shapes/
                    bool.cljs
                    circle.cljs
                    frame.cljs
                    group.cljs
                    image.cljs
                    multiple.cljs
                    path.cljs
                    rect.cljs
                    svg_raw.cljs
                    text.cljs

• In shapes/*.cljs there are the components that show the edit menu of each shape type.

• In menus/*.cljs there are the building blocks of these menus.

• And in rows/*.cljs there are some pieces, for example color input and picker.

Multiple edit #

When modifying the shape edit forms, you must take into account that these forms may edit several shapes at the same time, even of different types.

When more than one shape is selected, the form inside multiple.cljs is used. At the top of this module, a couple of maps define what attributes may be edited and how, for each type of shape.

Then, the blocks in menus/*.cljs are used, but they are not given a shape, but a values map. For each attribute, if all shapes have the same value, it is taken; if not, the attribute will have the value :multiple.

The form blocks must be prepared for this value, display something useful to the user in this case, and do a meaningful action when changing the value. Usually this will be to set the attribute to a fixed value in all selected shapes, but only those that may have the attribute (for example, only text shapes have font attributes, or only rects has border radius).

Component synchronization #

▾ common/
  ▾ src/app/common/
    ▾ types/
        component.cljc

For all shape attributes, you must take into account what happens when the attribute in a main component is changed and then the copies are synchronized.

In component.cljc there is a structure sync-attrs that maps shape attributes to sync groups. When an attribute is changed in a main component, the change will be propagated to its copies. If the change occurs in a copy, the group will be marked as touched in the copy shape, and from then on, further changes in the main to this attribute, or others in the same group, will not be propagated.

Any attribute that is not in this map will be ignored in synchronizations.

Render shapes, export & import #

▾ frontend/
  ▾ src/
    ▾ app/
      ▾ main/
        ▾ ui/
          ▾ shapes/
            ▸ text/
              attrs.cljs
              bool.cljs
              circle.cljs
              custom_stroke.cljs
              embed.cljs
              export.cljs
              fill_image.cljs
              filters.cljs
              frame.cljs
              gradients.cljs
              group.cljs
              image.cljs
              mask.cljs
              path.cljs
              rect.cljs
              shape.cljs
              svg_defs.cljs
              svg_raw.cljs
              text.cljs
      ▾ worker/
        ▾ import/
            parser.cljs

To export a penpot file, basically we use the same system that is used to display shapes in the workspace or viewer. In shapes/*.cljs there are components that render one shape of each type into a SVG node.

But to be able to import the file later, some attributes that not match directly to SVG properties need to be added as metadata (for example, proportion locks, constraints, stroke alignment...). This is done in the export.cljs module.

Finally, to import a file, we make use of parser.cljs, a module that contains the parse-data function. It receives a SVG node (possibly with children) and converts it into a Penpot shape object. There are auxiliary functions to read and convert each group of attributes, from the node properties or the metadata (with the get-meta function).

Any attribute that is not included in the export and import functions will not be exported and will be lost if reimporting the file again.

Code generation #

▾ frontend/
  ▾ src/
    ▾ app/
      ▾ main/
        ▾ ui/
          ▾ viewer/
            ▾ inspect/
              ▾ attributes/
                  blur.cljs
                  common.cljs
                  fill.cljs
                  image.cljs
                  layout.cljs
                  shadow.cljs
                  stroke.cljs
                  svg.cljs
                  text.cljs
                attributes.cljs
                code.cljs
      ▾ util/
          code_gen.cljs
          markup_html.cljs
          markup_svg.cljs
          style_css.cljs
          style_css_formats.cljs
          style_css_values.cljs

In the inspect panel we have two modes:

For the Info tab, the attributes.cljs module and all modules under attributes/*.cljs have the components that extract the attributes to inspect each type of shape.

For the Code tab, the util/code_gen.cljs module is in charge. It calls the other modules in util/ depending on the format.

For HTML and CSS, there are functions that generate the code as needed from the shapes. For SVG, it simply takes the nodes from the viewer main viewport and prettily formats it.