docs/dev/i18n.rst → docs/dev/web/i18n.rst

@@ -10,53 +10,70 @@ General guidelines
   nice-to-have, but not required. If you don't know a language well, rather leave the translation
   blank, so it is obvious that there is no proper translation yet.
 * Existing expert mode packages should be made translatable as soon as possible.
-* The "message IDs" (which are the arguments to the ``translate`` function) should be the
+* The "message IDs" (which are the arguments to the *translate* function) should be the
   English texts.
 
-i18n support in LuCI
---------------------
+i18n support in Gluon
+---------------------
 
-Internationalization support can be found in the ``luci.i18n`` package.
-Strings are translated using the ``i18n.translate`` and ``i18n.translatef`` functions
-(``translate`` for static strings, ``translatef`` for printf-like formatted string).
+Internationalization support is available in all components (models, view and
+controllers) of *gluon-web*-based packages. Strings are translated using the *translate*,
+*_translate* and *translatef* functions (*translate* for static strings, *translatef*
+for printf-like formatted string; *_translate* works the same as *translate*, but
+will return *nil* instead of the original string when no translation is available).
 
-Example from the ``gluon-config-mode-geo-location`` package::
+In views, the special tags ``<%:...%>`` can be used to translate the contained string.
 
-  local i18n = require "luci.i18n"
-  o = s:option(cbi.Flag, "_location", i18n.translate("Show node on the map"))
+Example from the *gluon-config-mode-geo-location* package:
+
+.. code-block:: lua
+
+  local share_location = s:option(Flag, "location", translate("Show node on the map"))
+
+Note that translations are *namespaced*: each package will only use its own
+translation strings by default. For this purpose, the package name must by
+specified in a package's controller. It is possible to access a different
+translation package using the *i18n* function from models and view, which is
+necessary when strings from the site configuration are used, or packages do not
+have their own controller (which is the case for config mode wizard components).
+
+.. code-block:: lua
+
+  local site_i18n = i18n 'gluon-site'
+  local msg = site_i18n._translate('gluon-config-mode:welcome')
 
 Adding translation templates to Gluon packages
 ----------------------------------------------
 
 The i18n support is based on the standard gettext system. For each translatable package,
-a translation template with extension ``.pot`` can be created using the ``i18n-scan.pl``
-script from the LuCI repository::
+a translation template with extension ``.pot`` can be created using the *i18n-scan.pl*
+script in the ``contrib`` directory:
 
-  cd package/gluon-config-mode-geo-location
+.. code-block:: sh
+
+  cd package/gluon-web-mesh-vpn-fastd
   mkdir i18n
   cd i18n
-  ../../../packages/luci/build/i18n-scan.pl ../files > gluon-config-mode-geo-location.pot
+  ../../../contrib/i18n-scan.pl ../files ../luasrc > gluon-web-mesh-vpn-fastd.pot
 
-The entries in the template can be reordered after the generation if desirable. Lots of standard
-translations like "Cancel" are already available in the LuCI base translation file (see
-``packages/luci/po/templates/base.pot``) and can be removed from the template.
+The same command can be run again to update the template.
 
-In addition, some additions to the Makefile must be made. Instead of OpenWrt's default ``package.mk``,
-the Gluon version ``$(GLUONDIR)/include/package.mk`` must be used. The i18n files must be installed
+In addition, the Makefile must be adjusted. Instead of OpenWrt's default *package.mk*,
+the Gluon version (``../gluon.mk`` for core packages) must be used. The i18n files must be installed
 and PKG_CONFIG_DEPENDS must be added::
 
   ...
-  include $(GLUONDIR)/include/package.mk
+  include ../gluon.mk
 
   PKG_CONFIG_DEPENDS += $(GLUON_I18N_CONFIG)
   ...
   define Build/Compile
-    $(call GluonBuildI18N,gluon-config-mode-geo-location,i18n)
+    $(call GluonBuildI18N,gluon-web-mesh-vpn-fastd,i18n)
   endef
 
-  define Package/gluon-config-mode-geo-location/install
+  define Package/gluon-web-mesh-vpn-fastd/install
     ...
-    $(call GluonInstallI18N,gluon-config-mode-geo-location,$(1))
+    $(call GluonInstallI18N,gluon-web-mesh-vpn-fastd,$(1))
   endef
   ...
 
@@ -64,29 +81,29 @@ and PKG_CONFIG_DEPENDS must be added::
 Adding translations
 -------------------
 
-A new translation file for a template can be added using the ``msginit`` command::
+A new translation file for a template can be added using the *msginit* command:
+
+.. code-block:: sh
 
-  cd package/gluon-config-mode-geo-location/i18n
+  cd package/gluon-web-mesh-vpn-fastd/i18n
   msginit -l de
 
-This will create the file ``de.po`` in which the translations can be added.
+This will create the file *de.po* in which the translations can be added.
 
-The translation file can be updated to a new template version using the ``msgmerge`` command::
+The translation file can be updated to a new template version using the *msgmerge* command:
 
-  msgmerge -U de.po gluon-config-mode-geo-location.pot
+.. code-block:: sh
+
+  msgmerge -U de.po gluon-web-mesh-vpn-fastd.pot
 
 After the merge, the translation file should be checked for "fuzzy matched" entries where
 the original English texts have changed. All entries from the translation file should be
-translated in the ``.po`` file (or removed from it, so the original English texts are displayed
+translated in the *.po* file (or removed from it, so the original English texts are displayed
 instead).
 
 Adding support for new languages
 --------------------------------
 
-A list of all languages supported by LuCI can be found in the ``packages/luci/luci.mk`` file after
-Gluon's dependencies have been downloaded using ``make update``. Adding translations for these
-languages is straightforward using the ``msginit`` command.
-
-For other languages, support must be added to LuCI first, which constitutes completely translating
-the ``base.pot``. Please contact the upstream LuCI maintainers at https://github.com/openwrt/luci/
-if you'd like to do this.
+A list of all languages supported by *gluon-web* can be found in ``package/gluon.mk``.
+New languages just need to be added to *GLUON_SUPPORTED_LANGS*, and a human-readable
+language name must be defined.

docs/dev/web/model.rst

+Models
+======
+
+Models are defined in the ``model`` subdirectory of a gluon-web application
+(``/lib/gluon/config-mode/model`` for the config mode; the status
+page does not use any models). Model support is not part of the gluon-web core
+anymore, but is provided by the *gluon-web-model* package.
+
+Each model defines one or more forms to display on a page, and how the submitted
+form data is handled.
+
+Let's start with an example:
+
+.. code-block:: lua
+
+  local f = Form(translate('Hostname'))
+
+  local s = f:section(Section)
+
+  local o = s:option(Value, 'hostname', translate('Hostname'))
+  o.default = uci:get_first('system', 'system', 'hostname')
+  function o:write(data)
+    uci:set('system', uci:get_first('system', 'system'), 'hostname', data)
+    uci:commit('system')
+  end
+
+  return f
+
+The top-level element of a model is always a *Form*, but it is also possible for
+a model to return multiple forms, which are displayed one below the other.
+
+A *Form* has one or more *Sections*, and each *Section* has different types
+of options.
+
+All of these elements have an *id*, which is used to identify them in the HTML
+form and handlers. If no ID is given, numerical IDs will be assigned automatically,
+but using explicitly named elements is often advisable (and it is required if a
+form does not always include the same elements, i.e., some forms, sections or
+options are added conditionally). IDs are hierarchical, so in the above example,
+the *Value* would get the ID ``1.1.hostname`` (value *hostname* in first section
+of first form).
+
+Classes and methods
+-------------------
+
+  - *Form* (*title*, *description*, *id*)
+
+    - *Form:section* (*type*, *title*, *description*, *id*)
+
+      Creates a new section of the given type (usually *Section*).
+
+    - *Form:write* ()
+
+      Is called after the form has been submitted (but only if the data is valid). It
+      is called last (after all options' *write* methods) and is usually used
+      to commit changed UCI packages.
+
+      The default implementation of *write* doesn't do anything, but it can be
+      overridden.
+
+  - *Section* (usually instantiated through *Form:section*)
+
+    - *Section:option* (*type*, *id*, *title*, *description*)
+
+      Creates a new option of the given type. Option types:
+
+        - *Value*: simple text entry
+        - *TextValue*: multiline text field
+        - *ListValue*: radio buttons or dropdown selection
+        - *DynamicList*: variable number of text entry fields
+        - *Flag*: checkbox
+
+Most option types share the same properties and methods:
+
+  - *default*: default value
+  - *optional*: value may be empty
+  - *datatype*: one of the types described in :ref:`web-model-datatypes`
+
+    By default (when *datatype* is *nil*), all values are accepted.
+
+  - *state*: has one of the values *FORM_NODATA*, *FORM_VALID* and *FORM_INVALID*
+    when read in a form handler
+
+    An option that has not been submitted because of its dependencies will have
+    the state *FORM_NODATA*, *FORM_INVALID* if the submitted value is not valid
+    according to the set *datatype*, and *FORM_VALID* otherwise.
+
+  - *data*: can be read in form handlers to get the submitted value
+
+  - *depends* (*self*, *option*, *value*): adds a dependency on another option
+
+    The option will only be shown when the passed option has the given value. This
+    is mainly useful when the other value is a *Flag* or *ListValue*.
+
+  - *depends* (*self*, *deps*): adds a dependency on multiple other options
+
+    *deps* must be a table with options as keys and values as values. The option
+    will only be shown when all passed options have the corresponding values.
+
+    Multiple alternative dependencies can be added by calling *depends* repeatedly.
+
+  - *value* (*self*, *value*, *text*): adds a choice to a *ListValue*
+
+  - *write* (*self*, *data*): is called with the submitted value when all form data is valid.
+
+    Does not do anything by default, but can be overridden.
+
+The *default* value, the *value* argument to *depends* and the output *data* always have
+the same type, which is usually a string (or *nil* for optional values). Exceptions
+are:
+
+  - *Flag* uses boolean values
+  - *DynamicList* uses a table of strings
+
+Despite its name, the *datatype* setting does not affect the returned value type,
+but only defines a validator the check the submitted value with.
+
+For a more complete example that actually makes use of most of these features,
+have a look at the model of the *gluon-web-network* package.
+
+.. _web-model-datatypes:
+
+Data types
+----------
+
+  - *integer*: an integral number
+  - *uinteger*: an integral number greater than or equal to zero
+  - *float*: a number
+  - *ufloat*: a number greater than or equal to zero
+  - *ipaddr*: an IPv4 or IPv6 address
+  - *ip4addr*: an IPv4 address
+  - *ip6addr*: an IPv6 address
+  - *wpakey*: a string usable as a WPA key (either between 8 and 63 characters, or 64 hex digits)
+  - *range* (*min*, *max*): a number in the given range (inclusive)
+  - *min* (*min*): a number greater than or equal to the given minimum
+  - *max* (*max*): a number less than or equal to the given maximum
+  - *irange* (*min*, *max*): an integral number in the given range (inclusive)
+  - *imin* (*min*): an integral number greater than or equal to the given minimum
+  - *imax* (*max*): an integral number less than or equal to the given maximum
+  - *minlength* (*min*): a string with the given minimum length
+  - *maxlength* (*max*): a string with the given maximum length
+
+Differences from LuCI
+---------------------
+
+  - LuCI's *SimpleForm* and *SimpleSection* are called *Form* and *Section*, respectively
+  - Is it not possible to add options to a *Form* directly, a *Section* must always
+    be created explicitly
+  - Many of LuCI's CBI classes have been removed, most importantly the *Map*
+  - The *rmempty* option attribute does not exist, use *optional* instead
+  - Only the described data types are supported
+  - Form handlers work completely differently (in particular, a *Form*'s *handle*
+    method should usually not be overridden in *gluon-web*)

docs/dev/web/view.rst

+Views
+=====
+
+The template parser reads views from the ``view`` subdirectory of a
+gluon-web application (``/lib/gluon/config-mode/view`` for the config mode,
+``lib/gluon/status-page/view`` for the status page).
+Writing own views should usually be avoided in favour of using :doc:`model`
+with their predefined views.
+
+Views are partial HTML pages, with additional template tags that allow
+to embed Lua code and translation strings. The following tags are defined:
+
+  - ``<%`` ... ``%>`` evaluates the enclosed Lua expression.
+  - ``<%|`` ... ``%>`` evaluates the enclosed Lua expression and prints its value.
+  - ``<%=`` ... ``%>`` evaluates the enclosed Lua expression and prints its value
+    *without escaping HTML entities*. This is useful when the value contains HTML code.
+  - ``<%+`` ... ``%>`` includes another template.
+  - ``<%:`` ... ``%>`` translates the enclosed string using the loaded i18n catalog.
+  - ``<%_`` ... ``%>`` translates the enclosed string *without escaping HTML entities*
+    in the translation. This only makes sense when the i18n catalog contains HTML code.
+  - ``<%#`` ... ``%>`` is a comment.
+
+All of these also come in the whitespace-stripping variants ``<%-`` and ``-%>`` that
+remove all whitespace before or after the tag.
+
+Complex combinations of HTML and Lua code are possible, for example:
+
+.. code-block:: text
+
+  <div>
+    <% if foo then %>
+      Content
+    <% end %>
+  </div>
+
+
+Variables and functions
+-----------------------
+
+Many call sites define additional variables (for example, model templates can
+access the model as *self* and a unique element ID as *id*), but the following
+variables and functions should always be available for the embedded Lua code:
+
+  - *renderer*: :ref:`web-controller-template-renderer`
+  - *http*: :ref:`web-controller-http`
+  - *request*: Table containing the path components of the current page
+  - *url* (*path*): returns the URL for the given path, which is passed as a table of path components.
+  - *attr* (*key*, *value*): Returns a string of the form ``key="value"``
+    (with a leading space character before the key).
+
+    *value* is converted to a string (tables are serialized as JSON) and HTML entities
+    are escaped. Returns an empty string when *value* is *nil* or *false*.
+  - *include* (*template*): Includes another template.
+  - *node* (*path*, ...): Returns the controller node for the given page (passed as
+    one argument per path component).
+
+    Use ``node(unpack(request))`` to get the node for the current page.
+  - *pcdata* (*str*): Escapes HTML entities in the passed string.
+  - *urlencode* (*str*): Escapes the passed string for use in an URL.
+  - *translate*, *_translate*, *translatef* and *i18n*: see :doc:`i18n`

docs/features/autoupdater.rst

@@ -7,30 +7,65 @@ Building Images
 ---------------
 
 By default, the autoupdater is disabled (as it is usually not helpful to have unexpected updates
-during development), but it can be enabled by setting the variable GLUON_BRANCH when building
-to override the default branch set in the set in the site configuration.
+during development), but it can be enabled by setting the variable ``GLUON_AUTOUPDATER_ENABLED`` to ``1`` when building.
+It is also possible to override the default branch during build using the variable ``GLUON_AUTOUPDATER_BRANCH``.
+
+If a default branch is set neither in *site.conf* nor via ``GLUON_AUTOUPDATER_BRANCH``, the default branch is
+implementation-defined. Currently, the branch with the first name in alphabetical order is chosen.
 
 A manifest file for the updater can be generated with `make manifest`. A signing script (using
-ecdsautils) can by found in the `contrib` directory. When creating the manifest, ``GLUON_PRIORITY`` can
-be set on the command line, or it can be taken from the ``site.mk``.
+``ecdsautils``) can be found in the `contrib` directory. When creating the manifest, the
+``PRIORITY`` value may be defined by setting ``GLUON_PRIORITY`` on the command line or in ``site.mk``.
+
+``GLUON_PRIORITY`` defines the maximum number of days that may pass between releasing an update and installation
+of the images. The update probability will start at 0 after the release time declared in the manifest file
+by the variable DATE and then slowly rise up to 1 when ``GLUON_PRIORITY`` days have passed. The autoupdater checks
+for updates hourly (at a random minute of the hour), but usually only updates during its run between
+4am and 5am, except when the whole ``GLUON_PRIORITY`` days and another 24 hours have passed.
+
+``GLUON_PRIORITY`` may be an integer or a decimal fraction.
+
+If ``GLUON_RELEASE`` is passed to ``make`` explicitly or it is generated dynamically
+in ``site.mk``, care must be taken to pass the same ``GLUON_RELEASE`` to ``make manifest``,
+as otherwise the generated manifest will be incomplete.
+
+
+Manifest format
+---------------
 
-The priority defines the maximum number of days that may pass between releasing an update and installation
-of the images. The update probability will start at 0 after the release time mentioned in the manifest
-and then slowly rise to 1 up to the point when the number of days given by the priority has passed.
+The manifest starts with a short header, followed by the list of firmwares and signatures.
+The header contains the following information:
 
-The priority may be an integer or a decimal fraction.
+.. code-block:: sh
+
+    BRANCH=stable
+    DATE=2020-10-07 00:00:00+02:00
+    PRIORITY=7
+
+- ``BRANCH`` is the autoupdater branch name that needs to match the nodes configuration.
+- ``DATE`` specifies when the time period for the update begins. Nodes will do their regular update during a random minute
+  between 4:00 and 4:59 am. Nodes might not always have a reliable NTP synchronization, which is why a fallback mechanism
+  exists, that checks for an update, and will execute if ``DATE`` is at least 24h in the past.
+- ``PRIORITY`` can be configured as ``GLUON_PRIORITY`` when generating the manifest or in ``site.mk``, and defines
+  the number of days over which the update should be stretched out after ``DATE``. Nodes will calculate a probability
+  based on the time left to determine when to update.
+
+
+Automated nightly builds
+------------------------
 
 A fully automated nightly build could use the following commands:
 
-::
+.. code-block:: sh
 
     git pull
-    (cd site && git pull)
+    # git -C site pull
     make update
-    make clean
+    make clean GLUON_TARGET=ath79-generic
     NUM_CORES_PLUS_ONE=$(expr $(nproc) + 1)
-    make -j$NUM_CORES_PLUS_ONE GLUON_TARGET=ar71xx-generic GLUON_BRANCH=experimental
-    make manifest GLUON_BRANCH=experimental
+    make -j$NUM_CORES_PLUS_ONE GLUON_TARGET=ath79-generic GLUON_RELEASE=$GLUON_RELEASE \
+        GLUON_AUTOUPDATER_BRANCH=experimental GLUON_AUTOUPDATER_ENABLED=1
+    make manifest GLUON_RELEASE=$GLUON_RELEASE GLUON_AUTOUPDATER_BRANCH=experimental
     contrib/sign.sh $SECRETKEY output/images/sysupgrade/experimental.manifest
 
     rm -rf /where/to/put/this/experimental
@@ -74,8 +109,6 @@ These commands can be used on a node:
 
 ::
 
-   # If fallback is true the updater will perform an update only if 
-   # the timespan given by the priority and another 24h have passed
+  # If fallback is true the updater will perform an update only if the timespan
+  # PRIORITY days (as defined in the manifest) and another 24h have passed
   autoupdater --fallback
-
-

docs/features/configmode.rst

@@ -14,10 +14,13 @@ Activating Config Mode
 ----------------------
 
 Config Mode is automatically entered at the first boot. You can re-enter
-Config Mode by pressing and holding the RESET/WPS button for about three
-seconds. The device should reboot (all LEDs will turn of briefly) and 
+Config Mode by pressing and holding the RESET/WPS/DECT button for about three
+seconds. The device should reboot (all LEDs will turn off briefly) and
 Config Mode will be available.
 
+If you have access to the console of the node, there is the
+``gluon-enter-setup-mode`` command, which reboots a node into Config Mode.
+
 
 Port Configuration
 ------------------
@@ -35,3 +38,17 @@ Accessing Config Mode
 Config Mode can be accessed at http://192.168.1.1. The node will offer DHCP
 to clients. Should this fail, you may assign an IP from 192.168.1.0/24 to
 your computer manually.
+
+.. image:: configmode.png
+
+Advanced Config Options
+-----------------------
+
+Depending on the installed packages, the advanced config mode allows to configure packages further.
+
+* :doc:`gluon-web-wifi-config enable <wlan-configuration>` radios used for wifi and mesh as well as outdoor mode
+* :doc:`../package/gluon-web-network` allows to configure the used roles (uplink, mesh, client) on each interface
+* :doc:`../package/gluon-web-admin` allows to enter SSH keys or set a password in the `Remote access` section
+* :doc:`../package/gluon-web-cellular` allows to configure SIM card / WWAN settings on supported cellular devices
+
+The advanced config does also allow to upload a sysupgrade file to update the firmware to a different version.

docs/features/dns-cache.rst

+.. _dns-caching:
+
+DNS caching
+===========
+
+User experience may be greatly improved when dns is accelerated. Also, it
+seems like a good idea to keep the number of packages being exchanged
+between node and gateway as small as possible. In order to do this, a
+DNS cache may be used on a node. The dnsmasq instance listening on port
+53 on the node will be reconfigured to answer requests, use a list of
+upstream servers and a specific cache size if the options listed below are
+added to site.conf. Upstream servers are the DNS servers which are normally
+used by the nodes to resolve hostnames (e.g. gateways/supernodes).
+
+There are the following settings:
+    servers
+    cacheentries
+
+To use the node's DNS server, both options should be set. The node will cache at
+most 'cacheentries' many DNS records in RAM. The 'servers' list will be used to
+resolve the received DNS queries if the request cannot be answered from
+cache. Gateways should announce the "next node" address via DHCP and RDNSS (if
+any). Note that not setting 'servers' here will lead to DNS not working: Once
+the gateways all announce the "next node" address for DNS, there is no way for
+nodes to automatically determine DNS servers. They have to be baked into the
+firmware.
+
+If these settings do not exist, the cache is not initialized and RAM usage will
+not increase.
+
+When next_node.name is set, an A record and an AAAA record for the
+next-node IP address are placed in the dnsmasq configuration. This means that
+the content of next_node.name may be resolved even without upstream connectivity.
+It is suggested to use the same name as the DNS server provides:
+e.g. nextnode.location.community.example.org (This way the name also works if a
+client uses static DNS Servers). Hint: If next_node.name does not contain a dot
+some browsers would open the searchpage instead.
+
+::
+
+  dns = {
+    cacheentries = 5000,
+    servers = { '2001:4860:4860::8888', '2001:4860:4860::8844' },
+  },
+
+  next_node = {
+    name = { 'nextnode.location.community.example.org', 'nextnode', 'nn' },
+    ip6 = '2001:db8:8::1',
+    ip4 = '198.51.100.1',
+  }
+
+
+Each cache entry will occupy about 90 bytes of RAM.