Improve IMAP and SENDER management

I changed my email provider recently and it seems I'm less free to set
the sender field now. So I cannot say this email comes from the real
user, I need to set it to the mailing list email, maybe the true email
behind the lists. This change allow this.

- Add some documentation (I did not updated mailinglistrb since some
  time so I needed to read my code again to understand what it does.
  This documentation should help any other user when configuring there
  own instance.
- Ignore more .env files, just in case
- Improve SMTP error handler (it does not do a lot of things but
  ensure we log unknown smtp errors as well)
- Add IMAP disconnection error management (it is still under test, I
  do not know for sure it fixed the issue where IMAP disconnected and
  not recovered. Also it is a bit clumsy as it does not ensure this at
  every step using the imap tcp socket.
- Add a SENDER config variable to force set sender field to something
  else than the original sender when distributing emails.
- Fix (probably ?) a bug where the port may not be taken into accound
  and ssl parameter neither. Did not tested it because I'm lazy.
- Replace mailinglist address suffix with + rather than . because it's
  probably a more common way to handle alias.
- Add more tasks to the Rakefile

.gitignore

@@ -1,5 +1,6 @@
-.env
+.env*
 *.db
 *.sqlite*
 README.html
 .yardoc/
+bin/db_seed.local

Gemfile

@@ -3,19 +3,19 @@
 source "https://rubygems.org"
 
 # Mails proto
-gem "net-smtp", "~> 0.3.3"
 gem "net-imap", "~> 0.3.1"
+gem "net-smtp", "~> 0.3.3"
 
 # Mail template
 gem "erb", "~> 3.0"
 
 # Database
-gem "sqlite3", "~> 1.5"
 gem "sequel", "~> 5.62"
+gem "sqlite3", "~> 1.5"
 
 # Web
-gem "sinatra", "~> 3.0"
 gem "puma", "~> 6.0"
+gem "sinatra", "~> 3.0"
 gem "slim", "~> 4.1"
 
 # Others
@@ -24,6 +24,6 @@ gem "uuid", "~> 2.3"
 
 # Debug and stuff
 group :develop do
-  gem "pry", "~> 0.14.1"
   gem "mocha", "~> 2.0"
+  gem "pry", "~> 0.14.1"
 end

README.md

@@ -37,6 +37,29 @@ Copy and fill all the env variables in .env
     cp empty.env .env
     edit .env
 
+Here is the list of all the configuration available (sampled in empty.env):
+
+    SMTP_HOST      information to connect on the smtp
+    SMTP_PORT
+    SMTP_TLS       true or false
+    SMTP_USER
+    SMTP_PASSWORD
+    IMAP_HOST      only imap available
+    IMAP_PORT
+    IMAP_TLS
+    IMAP_USER
+    IMAP_PASSWORD
+    SENDER_HOST    the domain of the sender field, and the mailinglist addresses
+    SENDER         "true" to retrive the true sender, "list" to use the mailing list email
+                   else it is a static email address that will alway be used.
+    DB_URL         sqlite://db/database.sqlite for instance
+    PORT           for WWW accees, not used yet
+    DEBUG          true/false
+    CPU_SLEEP      slow down the distributor
+    LOG_FILE       log stuff in the specified file
+
+
+
 ### Setup the database
 
     bin/db_migrate
@@ -53,6 +76,10 @@ After deploying it, there are some tools:
 
 - `bin/db_seed` to generate some data
 
+### Additional resources
+
+[How to manage mailinglists](doc/manage_mailinglists.md)
+
 ## Contributing
 
 ### via Gitea

Rakefile

@@ -7,6 +7,34 @@ Minitest::TestTask.create(:test) do |t|
   t.test_globs = ["test/**/test_*.rb"]
 end
 
+namespace "db" do
+  desc "Migrate the database to the lasted schema"
+  task "migrate" do
+    load "bin/db_migrate"
+  end
+
+  desc "Initialize database with dumby data"
+  task "dumb_seed" do
+    load "bin/db_seed"
+  end
+
+  desc "Initialize database with local data"
+  task "local_seed" do
+    load "bin/db_seed.local"
+  end
+
+  desc "Reset all tables, schema, data"
+  task "reset" do
+    require_relative "lib/app"
+    $db.tables.each { $db.drop_table _1 }
+  end
+
+  namespace "reset" do
+    task "stuff" do
+    end
+  end
+end
+
 task :default => :test
 
 task default: :test

doc/manage_mailinglists.md

@@ -0,0 +1,43 @@
+# Mailinglist management
+
+## What are strategies?
+
+There are 2 types of strategies: registration and moderation.
+Registration defines how new user can access the list.
+Moderation defines how user can write on the list.
+
+### Registration
+
+- autoregister: writing any email to the mailinglist will add the email of the user to the mailinglist
+- free: any email can register to the list by sending a "subscribe" email to the list
+- validated: like free but moderators needs to confirm the email with a "set-permissions" email. the user details are distributed to moderators first.
+- closed: nobody can register via email
+
+### Moderation
+
+- freewrite: anyone registered can distribute emails to the list
+- restrictedwrite: email needs approval by moderator before distribution. the email details are only distributed to moderators first.
+
+## How to interact with the list by email ?
+
+You need to send an email to the list with the subject being an action.
+Some action can take parameters in the subject or the body.
+
+Actions are:
+- set-permissions
+- validate
+- refuse
+- subscribe
+- unsubscribe
+- help
+- list-users
+- owner
+
+Parameters are key=value, which are separated with "," in the subject or "new line" in the body.
+
+### permissions parameters
+permissions: a permission mask (0=reader, 1=writer, 2=operator, 4=moderator)
+user-email: the email of the user
+
+### validate parameters
+uid: uid of the email

empty.env

@@ -11,6 +11,7 @@ IMAP_USER=
 IMAP_PASSWORD=
 
 SENDER_HOST=machin.fr
+SENDER=list@machin.fr
 DB_URL="sqlite://dev.db"
 
 PORT=10081

lib/models/mailinglist.rb

@@ -1,7 +1,7 @@
 require "uuid"
 
 class Mailinglist < Sequel::Model($db)
-  SUFFIX_SEPARATOR = ENV["MAILINGLIST_SUFFIX_SEPARATOR"] || "."
+  SUFFIX_SEPARATOR = ENV["MAILINGLIST_SUFFIX_SEPARATOR"] || "+"
   BASE_USER = ENV["MAILINGLIST_BASE_USER"] || "mailinglist"
   HOST = ENV["MAILINGLIST_HOST"]
 

lib/protocols/imap.rb

@@ -17,7 +17,7 @@ class Protocols::Imap
   attr_reader :imap if $debug
 
   def reset!
-    @imap = Net::IMAP.new(ENV["IMAP_HOST"], ENV["IMAP_PORT"], ssl: true)
+    @imap = Net::IMAP.new(ENV["IMAP_HOST"], port: ENV["IMAP_PORT"], ssl: true)
     @imap.authenticate('PLAIN', ENV["IMAP_USER"], ENV["IMAP_PASSWORD"])
     ALL_INBOXES.each do
       $logger.info "Try to create inbox #{_1}"
@@ -31,10 +31,18 @@ class Protocols::Imap
   # Fetch the next incomming email as a Protocols::Mail.
   #
   # @return [Protocols::Mail]
-  def fetch_next_unseen(inbox:)
-    uid = search_unseen(inbox:).first
-    return nil if uid.nil?
+  def fetch_next_unseen(inbox:, max_tries: 2)
+    return $logger.error("fetch_next_unseen reached max_tries_limit") if max_tries == 0
 
+    begin
+      uid = search_unseen(inbox:).first
+    rescue IOError => e
+      $logger.warn e.message
+      reset!
+      return fetch_next_unseen(inbox:, max_tries: max_tries - 1)
+    end
+
+    return nil if uid.nil?
     fetch(uid:, inbox:)
   end
 

lib/protocols/mail.rb

@@ -11,8 +11,10 @@ class Protocols::Mail
   HEADERS_KEEP = %w[
     Content-Type Content-Transfer-Encoding MIME-Version
   ].freeze
-  HEADERS_KEEP_WITH_ORIGINAL = HEADERS_KEEP + %w[Message-ID User-Agent From To Subject Subject Date In-Reply-To References]
+  HEADERS_KEEP_WITH_ORIGINAL = HEADERS_KEEP + %w[Message-ID User-Agent From To Subject Subject Date In-Reply-To
+                                                 References]
   USER_AGENT = "Ruby/#{RUBY_VERSION} (#{RUBY_PLATFORM}) Mailinglist.rb/1.0".freeze
+  SENDER = ENV["SENDER"]
 
   # @params imap_mail is any element outputed in the returned array of
   #   Net::IMAP#fetch, and must contains some attributes:
@@ -72,7 +74,16 @@ class Protocols::Mail
       ["Errors-To",        list.email],
       ["To",               dest.email],
       ["From",             "\"#{@from_name}\" (via #{list.name}) <#{@from}>"],
-      ["Sender",           @from],
+      [
+        "Sender",
+        if SENDER == "true"
+          @from
+        elsif SENDER == "list"
+          list.email
+        else
+          SENDER
+        end
+      ],
       ["Date",             Time.now.strftime(DATE_FORMAT)],
       ["List-Id",          "<#{list.email}>"],
       ["List-Post",        "<mailto:#{list.email}>"],

lib/protocols/smtp.rb

@@ -61,6 +61,9 @@ class Protocols::Smtp
       $logger.warn e.message
       reset_smtp_client!
       send_message_safe(*raw, max_tries: max_tries - 1)
+    rescue => e
+      $logger.error e.full_message
+      reset_smtp_client!
     end
   end
 end